Benchmark
A shared test that lets you compare systems. Useful, and easy to over-trust.
A benchmark is a dataset and a scoring method that many systems are measured against, which makes comparison possible. Public benchmarks are how progress gets tracked at all.
They also get gamed. Once a benchmark is famous, it becomes a training target, and the score stops measuring general capability. Numbers on a leaderboard also hide deployment details: a different prompt, a retrieval setup, or a pass rate on a task nobody in your organisation has.
Use benchmarks to narrow the field, not to decide. A public score can tell you a model is broadly capable enough to try. Only your own evaluation on your own data can tell you whether it is good enough for your use.
Related terms
Evaluation
Measuring whether the system does the job, on a fixed set of cases, repeatedly.
Golden Dataset
A fixed, reviewed set of real cases the system is measured against.
LLM Judge
Using a model to score another model's output against a rubric.
Model
The trained file of learned parameters that turns input into output.
Want this working inside your team?
It starts with one conversation about your workflow.