A practical guide to building an eval you can actually trust
You tweak a prompt. You run it on five examples. The outputs feel better, so you ship it. Two weeks later, a user points out that a whole category of questions has been quietly broken ever since.
This is the vibes problem, and almost everyone building with LLMs hits it. The fix is a proper evaluation. But evaluating LLMs is harder than evaluating classic machine learning models: outputs are open-ended, there is rarely a single correct answer, and the same input can produce different outputs on different runs.
The uncomfortable truth is that most bad evals fail not because the model is bad, but because the measurement is bad. A weak test set or an unvalidated LLM judge gives you confident-looking numbers that don’t mean what you think they mean. This post covers how to build a test set that reflects reality, how to use LLM-as-judge responsibly, and the specific ways these setups fool you.
Before you build anything, write down what “good” means for your application. Typical dimensions include:
It also helps to separate two kinds of evals. Capability evals ask “can the system do this at all?” and are usually hard. Regression evals ask “did my change make anything worse?” and should mostly pass. You need both, and they serve different purposes.
Finally, tie every metric to a decision. If a number moves and you wouldn’t change anything, it’s not worth tracking.
Your test set is the foundation. A flawed one makes everything built on top of it meaningless.
A hundred well-chosen examples beat a thousand near-duplicates. Make sure you cover:
Tag each example with metadata (intent, difficulty, source) so you can slice results later. An overall score of 85% can hide a 40% score on the one category that matters most.
Decide how “correct” is defined for each example:
Write annotation guidelines, have at least two people label an overlapping subset, and measure how often they agree. If humans can’t agree, a judge model won’t magically fix that. It usually means your criteria are ambiguous.
Smaller than you fear, larger than you’d like. The key is to think in terms of uncertainty. With 50 examples, a score of 80% has a wide confidence interval, so a “3-point improvement” is almost certainly noise. Start with 50 to 200 carefully chosen examples to catch big problems, and grow the set when you need to detect smaller differences. Always report a confidence interval, or at least be honest about the margin.
Before reaching for a judge model, ask whether the property can be checked mechanically:
Deterministic checks are cheap, fast, reproducible, and can’t be charmed by confident prose. Use them as your first layer and save the judge for what truly requires judgment.
LLM-as-judge means using a language model to grade another model’s output against a rubric. The common setups are:
It’s attractive because it scales: you can grade thousands of outputs in minutes, on qualities like relevance, helpfulness, and instruction following that are hard to capture with rules. Used carefully, it’s a genuinely useful tool. Used carelessly, it’s a machine for generating false confidence.
Each of these is well documented, and each can silently distort your results.
Position bias. In pairwise comparisons, judges often favor the answer shown first (or sometimes second), regardless of quality. If you always put your new model’s output in slot A, you may be measuring slot preference.
Verbosity bias. Longer answers tend to score higher, even when the extra length adds nothing. If your new prompt makes outputs wordier, your scores may rise while quality doesn’t.
Self-preference bias. Models can rate outputs from their own family more favorably. Using the same model to generate and judge is a conflict of interest.
Style over substance. Confident, well-formatted, fluent prose beats plain but correct answers. A polished hallucination can outscore a terse truth.
Score compression. Ask for a rating from 1 to 10 and you’ll often get a cluster of 7s and 8s. The scale loses the resolution you need to tell good from great.
Vague rubrics. “Rate the overall quality” invites noise. The judge is guessing at what you mean, and so are you.
Can’t verify what it doesn’t know. A judge without access to ground truth can’t reliably catch subtle factual errors. It may just agree that a claim sounds plausible.
Instability. Small changes to the judge prompt, the model version, or even sampling randomness can shift scores. A change in your metric that comes from the judge changing is not a change in your product.
Goodhart’s law. Once you optimize a system against a fixed judge, the system learns to please the judge, which isn’t the same as pleasing users. Any metric that becomes a target stops being a good metric.
None of the above means you shouldn’t use LLM judges. It means you should validate them like any other measurement instrument.
This is the single most important step. Hand-label a sample (around 100 examples is a good starting point), run your judge on the same examples, and measure agreement. Use a confusion matrix for pass/fail judgments, or correlation and Cohen’s kappa when you have scales. Look at where they disagree, not just how often. If agreement is poor, fix the judge before using it for anything.
Prefer narrow, concrete criteria over broad ones, and give the judge small scales or binary decisions. For example:
You are grading a customer support answer.
Question: {question}
Context (source of truth): {context}
Answer: {answer}
Check each criterion:
1. Grounded: every factual claim is supported by the context. (yes/no)
2. Complete: the answer addresses all parts of the question. (yes/no)
3. Safe: the answer contains no unsupported promises or advice. (yes/no)
Explain your reasoning briefly for each criterion, then output JSON:
{‘grounded’: bool, ‘complete’: bool, ‘safe’: bool}
Three focused yes/no checks are usually more reliable and more debuggable than one “quality score from 1 to 10.” You can see which dimension regressed.
Asking the judge to explain its reasoning before giving a verdict often improves consistency, and it gives you something to read when you investigate disagreements. Test whether it helps on your task rather than assuming.
Here’s a minimal sketch of position-swapped pairwise judging and a bootstrap confidence interval:
import random
def judge_pair(a, b, question):
# returns 'A' or 'B' from your judge call
...
def robust_preference(out_new, out_old, question):
first = judge_pair(out_new, out_old, question) # new is A
second = judge_pair(out_old, out_new, question) # new is B
new_wins = (first == 'A') + (second == 'B')
if new_wins == 2: return 'new'
if new_wins == 0: return 'old'
return 'tie' # inconsistent verdicts count as ties
def bootstrap_ci(scores, n=2000, alpha=0.05):
means = []
for _ in range(n):
sample = [random.choice(scores) for _ in scores]
means.append(sum(sample) / len(sample))
means.sort()
return means[int(n * alpha / 2)], means[int(n * (1 - alpha / 2))]Run the judge several times on the same outputs to see how much it wanders. Report uncertainty, not just a single number. And re-check judge-versus-human agreement whenever you change the judge model, the judge prompt, or the distribution of your data.
Putting it all together:
Before you trust a number, ask yourself:
You don’t need to build everything from scratch. Open-source and hosted options such as promptfoo, DeepEval, Ragas, LangSmith, and Braintrust offer test-set management, judge templates, and experiment tracking. They save real time, but they can’t fix a bad test set or an uncalibrated judge. Choose based on how well they fit your workflow, and keep your own data and labels portable.
A trustworthy LLM evaluation has three ingredients: a test set that mirrors real usage, a judge you’ve validated against human judgment, and honesty about uncertainty. None of them requires a huge budget or a sophisticated platform. They require discipline: looking at your data, questioning your metrics, and refusing to believe a number just because it’s convenient.
If you do one thing this week, collect 50 real examples from your application, label them carefully, and check your judge against them. You’ll almost certainly learn something that no leaderboard or vibes check would have told you.
Introduction: Why Duplicate Detection Matters in Python Data quality plays a crucial role in data…
Introduction: Why BM25 Matters Every search begins with a question: how do we find the…
Introduction How a computer represents language has a major impact on what it can learn…
Introduction: When Language Models Get Confused Language models have become remarkably good at understanding and…
Introduction: The Shift from Models to Agents Over the past decade, Natural Language Processing (NLP)…
Introduction Large Language Models (LLMs) have rapidly become a core component of modern applications, powering…