Data Science

LLM Evaluation Without Fooling Yourself: Test Sets and LLM-as-Judge

A practical guide to building an eval you can actually trust

You tweak a prompt. You run it on five examples. The outputs feel better, so you ship it. Two weeks later, a user points out that a whole category of questions has been quietly broken ever since.

This is the vibes problem, and almost everyone building with LLMs hits it. The fix is a proper evaluation. But evaluating LLMs is harder than evaluating classic machine learning models: outputs are open-ended, there is rarely a single correct answer, and the same input can produce different outputs on different runs.

The uncomfortable truth is that most bad evals fail not because the model is bad, but because the measurement is bad. A weak test set or an unvalidated LLM judge gives you confident-looking numbers that don’t mean what you think they mean. This post covers how to build a test set that reflects reality, how to use LLM-as-judge responsibly, and the specific ways these setups fool you.

Decide what you’re measuring

Before you build anything, write down what “good” means for your application. Typical dimensions include:

  • Task success: did the output accomplish what the user needed?
  • Factuality or groundedness: is it correct, and supported by the provided context?
  • Format compliance: valid JSON, correct length, required fields present
  • Tone and style: matches your brand or the user’s expectations
  • Safety: refuses what it should, and doesn’t refuse what it shouldn’t

It also helps to separate two kinds of evals. Capability evals ask “can the system do this at all?” and are usually hard. Regression evals ask “did my change make anything worse?” and should mostly pass. You need both, and they serve different purposes.

Finally, tie every metric to a decision. If a number moves and you wouldn’t change anything, it’s not worth tracking.

Build a test set that reflects reality

Your test set is the foundation. A flawed one makes everything built on top of it meaningless.

Where examples come from

  • Production logs. The best source, because they show what users really ask. Handle privacy and PII carefully, and remove or anonymize anything sensitive.
  • Expert-written cases. Domain experts can write the hard, high-stakes examples that rarely show up in logs.
  • Synthetic data. An LLM can generate variations and fill coverage gaps quickly. Treat it as a supplement, not a replacement: synthetic examples tend to be cleaner, more uniform, and easier than real traffic.

Optimise for coverage, not volume

A hundred well-chosen examples beat a thousand near-duplicates. Make sure you cover:

  • the main user intents, weighted roughly by how often they occur
  • different difficulty levels, from trivial to genuinely hard
  • edge cases: very short or very long inputs, typos, multiple languages, ambiguous requests
  • adversarial inputs such as prompt injection attempts
  • “negative” cases where the right behaviour is to refuse, ask a clarifying question, or say “I don’t know”

Tag each example with metadata (intent, difficulty, source) so you can slice results later. An overall score of 85% can hide a 40% score on the one category that matters most.

Labelling: references, rubrics, or preferences

Decide how “correct” is defined for each example:

  • Reference answers work well when there’s a clear right answer.
  • Rubrics (a checklist of what a good answer must include or avoid) work better for open-ended tasks.
  • Pairwise preferences (which of two outputs is better) are easier for humans to give than absolute scores.

Write annotation guidelines, have at least two people label an overlapping subset, and measure how often they agree. If humans can’t agree, a judge model won’t magically fix that. It usually means your criteria are ambiguous.

How big is big enough?

Smaller than you fear, larger than you’d like. The key is to think in terms of uncertainty. With 50 examples, a score of 80% has a wide confidence interval, so a “3-point improvement” is almost certainly noise. Start with 50 to 200 carefully chosen examples to catch big problems, and grow the set when you need to detect smaller differences. Always report a confidence interval, or at least be honest about the margin.

Keep your test set clean

  • Separate your data. Use a development set to iterate on prompts and a held-out set you rarely touch to report results. If you tune against the same examples you report on, you’re measuring memorization of your own test.
  • Watch for contamination. Examples that leak into few-shot prompts, fine-tuning data, or retrieval indexes inflate your scores.
  • Version it. Treat the dataset like code. When you add or fix examples, record it, so scores from last month remain comparable.
  • Refresh it. User behaviour drifts. Feed new production failures back into the set.

Use deterministic checks wherever you can

Before reaching for a judge model, ask whether the property can be checked mechanically:

  • exact match or normalized string match
  • regex or schema validation for structured output
  • executing generated code against unit tests
  • retrieval metrics (such as recall@k) for the search part of a RAG system
  • checking that cited sources actually appear in the provided context

Deterministic checks are cheap, fast, reproducible, and can’t be charmed by confident prose. Use them as your first layer and save the judge for what truly requires judgment.

LLM-as-judge: what it is and when it helps

LLM-as-judge means using a language model to grade another model’s output against a rubric. The common setups are:

  • Single-answer grading: score one output, either pass/fail or on a small scale
  • Reference-guided grading: compare the output to a known good answer
  • Pairwise comparison: pick the better of two outputs (useful for A/B testing prompts or models)

It’s attractive because it scales: you can grade thousands of outputs in minutes, on qualities like relevance, helpfulness, and instruction following that are hard to capture with rules. Used carefully, it’s a genuinely useful tool. Used carelessly, it’s a machine for generating false confidence.

How LLM judges fool you

Each of these is well documented, and each can silently distort your results.

Position bias. In pairwise comparisons, judges often favor the answer shown first (or sometimes second), regardless of quality. If you always put your new model’s output in slot A, you may be measuring slot preference.

Verbosity bias. Longer answers tend to score higher, even when the extra length adds nothing. If your new prompt makes outputs wordier, your scores may rise while quality doesn’t.

Self-preference bias. Models can rate outputs from their own family more favorably. Using the same model to generate and judge is a conflict of interest.

Style over substance. Confident, well-formatted, fluent prose beats plain but correct answers. A polished hallucination can outscore a terse truth.

Score compression. Ask for a rating from 1 to 10 and you’ll often get a cluster of 7s and 8s. The scale loses the resolution you need to tell good from great.

Vague rubrics. “Rate the overall quality” invites noise. The judge is guessing at what you mean, and so are you.

Can’t verify what it doesn’t know. A judge without access to ground truth can’t reliably catch subtle factual errors. It may just agree that a claim sounds plausible.

Instability. Small changes to the judge prompt, the model version, or even sampling randomness can shift scores. A change in your metric that comes from the judge changing is not a change in your product.

Goodhart’s law. Once you optimize a system against a fixed judge, the system learns to please the judge, which isn’t the same as pleasing users. Any metric that becomes a target stops being a good metric.

Making your judge trustworthy

None of the above means you shouldn’t use LLM judges. It means you should validate them like any other measurement instrument.

Calibrate against human labels first

This is the single most important step. Hand-label a sample (around 100 examples is a good starting point), run your judge on the same examples, and measure agreement. Use a confusion matrix for pass/fail judgments, or correlation and Cohen’s kappa when you have scales. Look at where they disagree, not just how often. If agreement is poor, fix the judge before using it for anything.

Write specific rubrics

Prefer narrow, concrete criteria over broad ones, and give the judge small scales or binary decisions. For example:

You are grading a customer support answer.

Question: {question}
Context (source of truth): {context}
Answer: {answer}

Check each criterion:
1. Grounded: every factual claim is supported by the context. (yes/no)
2. Complete: the answer addresses all parts of the question. (yes/no)
3. Safe: the answer contains no unsupported promises or advice. (yes/no)

Explain your reasoning briefly for each criterion, then output JSON:
{‘grounded’: bool, ‘complete’: bool, ‘safe’: bool}

Three focused yes/no checks are usually more reliable and more debuggable than one “quality score from 1 to 10.” You can see which dimension regressed.

Let it reason, then score

Asking the judge to explain its reasoning before giving a verdict often improves consistency, and it gives you something to read when you investigate disagreements. Test whether it helps on your task rather than assuming.

Counter the known biases

  • Swap positions in pairwise comparisons and average the results (or only count verdicts that survive the swap).
  • Control for length. Instruct the judge not to reward verbosity, and check whether scores correlate with output length.
  • Use a different model family as judge than the one generating outputs.
  • Ground the judge by supplying reference answers or retrieved context, so it isn’t relying on its own memory.

Here’s a minimal sketch of position-swapped pairwise judging and a bootstrap confidence interval:

import random

def judge_pair(a, b, question):
    # returns 'A' or 'B' from your judge call
    ...

def robust_preference(out_new, out_old, question):
    first = judge_pair(out_new, out_old, question)    # new is A
    second = judge_pair(out_old, out_new, question)   # new is B
    new_wins = (first == 'A') + (second == 'B')
    if new_wins == 2: return 'new'
    if new_wins == 0: return 'old'
    return 'tie'  # inconsistent verdicts count as ties

def bootstrap_ci(scores, n=2000, alpha=0.05):
    means = []
    for _ in range(n):
        sample = [random.choice(scores) for _ in scores]
        means.append(sum(sample) / len(sample))
    means.sort()
    return means[int(n * alpha / 2)], means[int(n * (1 - alpha / 2))]

Measure variance and re-validate

Run the judge several times on the same outputs to see how much it wanders. Report uncertainty, not just a single number. And re-check judge-versus-human agreement whenever you change the judge model, the judge prompt, or the distribution of your data.

A practical eval workflow

Putting it all together:

  1. Define success for your application, in specific, decision-relevant terms.
  2. Collect a small, representative test set (50 to 200 examples) from real usage, plus hand-written edge cases.
  3. Add deterministic checks for everything that can be checked mechanically.
  4. Build your judge(s), then calibrate them against human labels before trusting them.
  5. Run evals automatically on every prompt or model change, ideally in CI.
  6. Read the outputs. Skim failures and a random sample of passes. This is the highest-return habit in evaluation, and no metric replaces it.
  7. Close the loop. Add production failures to your test set so the eval gets harder in the right ways.

Pitfall checklist

Before you trust a number, ask yourself:

  • Does my test set resemble real traffic, or just the easy cases?
  • Did I tune my prompt on the same examples I’m reporting on?
  • Is this improvement bigger than the noise? Do I have a confidence interval?
  • Have I actually read a sample of the outputs?
  • Is the judge from a different model family than the generator?
  • Have I checked the judge against human labels, and recently?
  • Did the new version simply get longer?
  • Am I slicing results by category, or only looking at the average?

Tools

You don’t need to build everything from scratch. Open-source and hosted options such as promptfoo, DeepEval, Ragas, LangSmith, and Braintrust offer test-set management, judge templates, and experiment tracking. They save real time, but they can’t fix a bad test set or an uncalibrated judge. Choose based on how well they fit your workflow, and keep your own data and labels portable.

Conclusion

A trustworthy LLM evaluation has three ingredients: a test set that mirrors real usage, a judge you’ve validated against human judgment, and honesty about uncertainty. None of them requires a huge budget or a sophisticated platform. They require discipline: looking at your data, questioning your metrics, and refusing to believe a number just because it’s convenient.

If you do one thing this week, collect 50 real examples from your application, label them carefully, and check your judge against them. You’ll almost certainly learn something that no leaderboard or vibes check would have told you.

Neri Van Otten

Neri Van Otten is the founder of Spot Intelligence, a machine learning engineer with over 12 years of experience specialising in Natural Language Processing (NLP) and deep learning innovation. Dedicated to making your projects succeed.

Recent Posts

Duplicate Detection in Python: Techniques, Libraries, and Comprehensive How-To Tutorials

Introduction: Why Duplicate Detection Matters in Python Data quality plays a crucial role in data…

7 days ago

BM25 — The Ranking Algorithm Behind Modern Search

Introduction: Why BM25 Matters Every search begins with a question: how do we find the…

4 weeks ago

Sparse vs Dense Representations in NLP: How Should NLP Systems Represent Language?

Introduction How a computer represents language has a major impact on what it can learn…

1 month ago

Adversarial Attacks on NLP Systems; How Fragile Are They?

Introduction: When Language Models Get Confused Language models have become remarkably good at understanding and…

2 months ago

Agentic NLP Systems: How Autonomous AI Agents Use Large Language Models

Introduction: The Shift from Models to Agents Over the past decade, Natural Language Processing (NLP)…

4 months ago

Prompt Injection Attacks: Risks And How To Defend

Introduction Large Language Models (LLMs) have rapidly become a core component of modern applications, powering…

5 months ago