LLMs being released almost weekly. Some recent releases we’ve had are Qwen3 coing models, GPT 5, Grok 4, all of which claim the top of some benchmarks. Common benchmarks are Humanities Last Exam, SWE-bench, IMO, and so on.
However, these benchmarks have an inherent flaw: The companies releasing new front-end models are strongly incentivized to optimize their models for such performance on these benchmarks. The reason is that these well-known benchmarks are essentially what set the standard for what’s considered a new breakthrough LLM.
Luckily, there exists a simple solution to this problem: Develop your own internal benchmarks, and test each LLM on the benchmark, which is what I’ll be discussing in this article.
Table of Contents
You can also learn about How to Benchmark LLMs – ARC AGI 3, or you can read about ensuring reliability in LLM applications.
Motivation
My motivation for this article is that new LLMs are released rapidly. It’s difficult to stay up to date on all advances within the LLM space, and you thus have to trust benchmarks and online opinions to figure out which models are best. However, this is a severely flawed approach to judging which LLMs you should use either day-to-day or in an application you are developing.
Benchmarks have the flaw that frontier model developers are incentivized to optimize their models for benchmarks, making benchmark performance possibly flawed. Online opinions also have their problems because others will have other use cases for LLMs than you. Thus, you should develop an internal benchmark to properly test newly released LLMs and figure out which ones work best for your specific use case.
How to develop an internal benchmark
There are many approaches to developing your own internal benchmark. The main point here is that your benchmark is not a super common task LLMs perform (generating summaries, for example, does not work). Furthermore, your benchmark should preferably utilize some internal data not available online.
You should keep two main things in mind when developing an internal benchmark
- It should be a task that’s either uncommon (so the LLMs are not specifically trained on it), or it should be using data not available online
- It should be as automatic as possible. You don’t have time to test each new release manually
- You get a numeric score from the benchmark so that you can rank different models against each other
Types of tasks
Internal benchmarks could look very different from each other. Given some use cases, here are some example benchmarks you can develop
Use case: Development in a rarely used programming language.
Benchmark: Have the LLM zero-shot a specific application like Solitaire (This is inspired by how Fireship benchmarks LLMs by developing a Svelte application)
Use case: Internal question answering chatbot
Benchmark: Gather a series of prompts from your application (preferably actual user prompts), together with their desired response, and see which LLM is closest to the desired responses.
Use case: Classification
Benchmark: Create a dataset of input output examples. For this benchmark, the input can be a text, and the output a specific label, such as a sentiment analysis dataset. Evaluation is simple in this case, since you need the LLM output to exactly match the ground truth label.
Ensuring automatic tasks
After figuring out which task you want to create internal benchmarks for, it’s time to develop the task. When developing, it’s important to ensure the task runs as automatically as possible. If you had to perform a lot of manual work for each new model release, it would be impossible to maintain this internal benchmark.
I thus recommend creating a standard interface for your benchmark, where the only thing you need to change per new model is to add a function that takes in the prompt and outputs the raw model text response. Then the rest of your application can remain static when new models are released.
To keep the evaluations as automated as possible, I recommend running automated evaluations. I recently wrote an article about How to Perform Comprehensive Large Scale LLM Validation, where you can learn more about automated validation and evaluation. The main highlights are that you can either run a Regex function to verify correctness or utilize LLM as a judge.
Testing on your internal benchmark
Now that you’ve developed your internal benchmark, it’s time to test some LLMs on it. I recommend at least testing out all closed-source frontier model developers, such as
However, I also highly recommend testing out open-source releases as well, for example, with
In general, whenever a new model makes a splash (for example, when DeepSeek released R1), I recommend running it on your benchmark. And because you made sure to develop your benchmark to be as automated as possible, the cost is low to test out new models.
Continuing, I also recommend paying attention to new model version releases. For example, Qwen initially released their Qwen 3 model. However, a while later, they updated this model with Qwen-3-2507, which is said to be an improvement over the baseline Qwen 3 model. You should make sure to stay up to date on such (smaller) model releases as well.
My final point on running the benchmark is that you should run the benchmark regularly. The reason for this is that models can change over time. For example, if you’re using OpenAI and not locking the model version, you can experience changes in outputs. It’s thus important to regularly run benchmarks, even on models you’ve already tested. This applies especially if you have such a model running in production, where maintaining high-quality outputs is critical.
Avoiding contamination
When utilizing an internal benchmark, it’s incredibly important to avoid contamination, for example, by having some of the data online. The reason for this is that today’s frontier models have essentially scraped the entire internet for web data, and thus, the models have access to all of this data. If your data is available online (especially if the solutions in your benchmarks are available), you’ve got a contamination issue at hand, and the model probably has access to the data from its pre-training.
Use as little time as possible
Imagine this task as staying up to date on model releases. Yes, it’s a super important part of your job; however, this is a part that you can spend little time on and still get a lot of value. I thus recommend minimizing the time you spend on these benchmarks. Whenever a new frontier model is released, you test the model against your benchmark and verify the results. If the new model achieves vastly improved results, you should consider changing models in your application or day-to-day life. However, if you only see a small incremental improvement, you should probably wait for more model releases. Keep in mind that when you should change the model depends on factors such as:
- How much time does it take to change models
- The cost difference between the old and the new model
- Latency
- …
Conclusion
In this article, I have discussed how you can develop an internal benchmark for testing all the LLM releases happening recently. Staying up to date on the best LLMs is difficult, especially when it comes to testing which LLM works best on your use case. Developing internal benchmarks makes this testing process a lot faster, which is why I highly recommend it to stay up to date on LLMs.
👉 Find me on socials:
🧑💻 Get in touch
✍️ Medium
Or read my other articles:
