The Benchmark Problem
Benchmarks like MMLU, HumanEval, and GSM8K are useful for comparing models on standardized tasks and tracking progress over time. They are not useful for deciding which model to use for your specific product. A model that scores higher on MMLU might perform worse on your customer support task because the benchmark measures general knowledge in a specific format, while your task requires domain-specific reasoning, specific output formats, and tolerance for ambiguous inputs.
This distinction - capability vs. fit - is fundamental. Benchmarks measure capability: what the model can do in the abstract. Products require fit: how the model performs on your specific task with your specific data distribution and quality requirements. The teams that make good model selection decisions are the ones that measure fit, not just capability.
Building a Task-Specific Evaluation Set
The starting point for reliable evaluation is a task-specific test set that reflects the actual distribution of inputs your application will handle. This means collecting real production examples - or carefully constructed proxies if you are early in development - and annotating them with correct outputs or evaluation criteria.
For extractive tasks like named entity recognition or question answering, evaluation is relatively straightforward: compare model output to ground truth with standard metrics like F1 or accuracy. For generative tasks like summarization or creative writing, evaluation is harder: you need either human ratings or reliable automated metrics that correlate with human judgment. LLM-based evaluators - using a model to judge outputs - have become practical for many tasks, though they introduce their own biases.
A minimum viable evaluation set for a production application is typically 100-500 examples, depending on the variability of your input distribution. The set should be refreshed periodically to track distribution shift as real-world usage evolves.
Evaluation Metrics That Matter
Beyond accuracy, the metrics that matter for production depend on your application. For customer-facing applications, tone and consistency matter as much as factual accuracy. For data extraction, exact match matters for structured fields while semantic accuracy suffices for free-text. For code generation, you care about whether the code is not just correct but also readable and maintainable.
Cost and latency are often underweighted in model evaluation. A model that scores 5% higher on accuracy but costs twice as much per call and runs twice as slowly may not be the right choice for your application. Holistic evaluation includes these factors alongside quality metrics.
Continuous Evaluation in Production
The most sophisticated teams run continuous evaluation in production: sampling real production outputs, routing them to human reviewers or automated evaluators, and tracking quality metrics over time. This catches distribution shift, model degradation after updates, and quality differences between model versions. A model that performed well in offline evaluation six months ago may perform worse today as the real-world input distribution has shifted or as the model has been updated.
Building an evaluation culture - treating evaluation as a continuous practice rather than a one-time selection decision - is what separates teams that maintain reliable AI products from those that experience silent quality degradation.