How to Evaluate LLMs: A Practical Guide to Testing AI Responses

To evaluate an LLM, you define what a good answer looks like for your task, build a test set that represents real usage, score outputs with the cheapest reliable grader available, and track the results over time. This guide walks through each step and ends with a small Python harness you can adapt.

The difficulty is that language models are not like ordinary functions. The same prompt can produce different valid answers, a fluent answer can be wrong, and a single average score hides the failures that matter. A leaderboard rank tells you little about whether a model will handle your support tickets, your documents, or your code.

After reading, you will be able to:

• Turn a vague goal (“the answers should be good”) into measurable criteria.
• Choose between code-based, human, and LLM-based grading.
• Use LLM-as-a-judge without being fooled by its known biases.
• Judge whether a pass rate from a small test set can be trusted.
• Run a minimal evaluation loop in plain Python.

What does it mean to evaluate an LLM?

Evaluation is a repeatable process for measuring how well a model, or the application built around it, performs against criteria you chose in advance. Two levels are worth separating:

• Model evaluation compares models on general capabilities, usually with public benchmarks.
• Application (system) evaluation tests your whole pipeline: prompt, retrieval, tools, and model, on inputs that look like your real traffic.

Public benchmarks help you shortlist models. Application evaluation tells you whether to ship. Most practitioners spend the majority of their effort on the second kind, and this guide focuses there.

Step 1: Define success criteria before you test

Start with the task, not the metric. Anthropic’s documentation on defining success criteria and building evaluations recommends criteria that are specific and measurable, and notes that most use cases need to be judged along several dimensions at once.

A weak criterion: “The summary is good.” A usable criterion: “The summary contains no claim absent from the source, is under 120 words, and mentions the deadline if one exists.”

The second version can be split into three checks, and two of them can be graded by code. Typical dimensions include:

Dimension	Question it answers	Example check
Correctness	Is the answer factually right?	Compare to a reference answer
Faithfulness	Does it stay within the provided source?	Claim-by-claim review
Format	Does it follow the required structure?	JSON schema validation
Instruction following	Did it do everything asked?	Required elements present
Tone and style	Does it fit the audience?	Rubric-based grading
Safety	Does it avoid disallowed content?	Policy rubric plus human review
Cost and latency	Is it affordable and fast enough?	Token and time thresholds

Write down what counts as failure, too. “Egregious” and “inconvenient” errors should not weigh the same.

Step 2: Build a test set that looks like real usage

Your evaluation is only as good as its inputs. A practical test set has these properties:

1. Task-specific. Sample from real or realistic requests rather than inventing only tidy examples. Anthropic’s guidance is to mirror your real task distribution and include edge cases.
2. Includes hard cases. Add ambiguous inputs, malformed inputs, off-topic requests, and cases where even people would disagree.
3. Tagged. Label each case (for example factual, format, refusal) so you can see where failures cluster instead of staring at one average.
4. Held out. Keep some cases out of prompt tuning. If you iterate on a prompt until every visible case passes, you have measured memorization of your own test set, not generalization.
5. Versioned. Store cases in a file under version control so a score change can be traced to a code change rather than a changed test set.

Starting with 20 to 50 carefully chosen cases is reasonable for a first pass. Grow the set whenever production reveals a failure you did not anticipate: every real bug should become a permanent test case.

Step 3: Choose how to grade each output

There are three families of graders, and the best evaluations combine them. The trade-offs below follow the framing in Anthropic’s evaluation documentation.

Grader	Strengths	Weaknesses	Best for
Code-based	Fast, cheap, deterministic, scalable	Rigid; misses nuance	Exact answers, formats, constraints, tests that run code
Human	Most flexible and highest quality	Slow, expensive, inconsistent between raters	Final validation, ambiguous cases, calibrating other graders
LLM-based	Scales to subjective criteria	Has biases; needs validation	Tone, helpfulness, coherence, rubric-style judgments

A good default rule: use code when you can, an LLM when you must, and humans to check both.

Code-based graders

Exact match is the simplest, but it is brittle: “Paris.” and “The capital of France is Paris” are both correct. Normalizing text and matching on whole words is usually better. Other deterministic checks include JSON validity, required keywords, forbidden phrases, length limits, regular expressions, and running generated code against unit tests.

Overlap metrics such as BLEU or ROUGE compare word overlap with a reference. They can be useful for narrow tasks, but they can score a correct paraphrase poorly and a fluent error well, so treat them as weak signals for open-ended generation.

Human grading

Humans remain the reference standard. To make human grading usable, give raters a written rubric, have at least a sample of cases rated by two people, and measure how often they agree. If humans cannot agree on a case, an automated grader will not resolve the ambiguity; the rubric needs work.

LLM-as-a-judge

Here a capable model scores another model’s output against a rubric. The foundational study is Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena by Zheng et al. (2023, arXiv:2306.05685, NeurIPS 2023 Datasets and Benchmarks Track). The authors report that strong judges such as GPT-4 reached over 80% agreement with human preferences, roughly the level at which humans agree with each other, on the benchmarks they studied.

The same paper documents limitations you must design around:

• Position bias: the judge favors an answer because of where it appears, and in their tests most judges favored the first position.
• Verbosity bias: longer answers win even when they are not better.
• Self-enhancement bias: a judge may prefer text produced by itself.
• Limited reasoning ability: judges can misgrade math and logic questions.

Those findings were measured on specific models and tasks in 2023, so do not assume your judge behaves identically. Measure it on your own data. Practical mitigations:

1. Write a narrow rubric with explicit scoring levels instead of “rate quality 1 to 10.”
2. Swap positions in pairwise comparisons and count a result only if the winner is the same in both orders.
3. Ask for reasoning before the score and then discard the reasoning, a tip also given in Anthropic’s documentation.
4. Provide a reference answer for correctness questions, and use code to grade anything that can be checked mechanically.
5. Use a different model family as judge when you are comparing outputs that include the judge’s own family.
6. Validate the judge against a small set of human-labeled examples before trusting it at scale.

Step 4: Evaluate more than the final answer

Many failures are invisible when you only read the final text. For applications built on more than a single prompt, add these layers:

• Retrieval (RAG systems): Did the retrieved passages contain the answer? Then check that the generated answer is supported by those passages, since a model can ignore good context or invent beyond it.
• Tool use and agents: Did the agent choose the right tool, pass valid arguments, and reach the correct end state? Grading the final state of the environment is often more reliable than grading the transcript.
• Consistency: Run each important case several times. If the answer flips between runs, a single pass/fail result is misleading.
• Robustness: Rephrase prompts, add typos, and change the order of information. A system that only works for one wording is fragile.
• Safety and refusals: Include inputs that should be refused and inputs that look risky but are legitimate, so you catch both under- and over-refusal.

Step 5: Trust your numbers: sample size and uncertainty

A pass rate is an estimate, not a fact. With few test cases it can be very noisy. A rough guide for a pass rate near 50%: the 95% margin of error is about 1.96 × √(0.25 / n).

Test cases (n)	Approximate 95% margin of error
25	±20 points
100	±10 points
400	±5 points

So if prompt A scores 72% and prompt B scores 76% on 100 cases, you cannot claim B is better; the difference is smaller than the noise. When comparing two versions, run both on the same cases and look at which individual cases changed, rather than only comparing averages. The harness below reports a bootstrap confidence interval to make this uncertainty visible.

A minimal evaluation harness in Python

The following script uses only the standard library. It defines test cases, grades them with code-based checks, prints pass rates by tag with a bootstrap interval, and demonstrates a position-consistency check for a pairwise judge.

Important: the “model” here is a stub with canned answers (two are wrong on purpose) so the output is reproducible. The “judge” is a deliberately biased toy function used only to demonstrate the test. Replace fake_model with a call to your real model; the grading and reporting code stays the same. I ran this script with Python 3.12 and the output shown below is from that run.

"""A tiny, dependency-free LLM evaluation harness (standard library only)."""
import json
import random
import re
from dataclasses import dataclass, field
from statistics import mean

# ---------- 1. Test cases ----------
@dataclass
class TestCase:
    id: str
    prompt: str
    reference: str | None = None          # expected answer, if one exists
    must_include: list[str] = field(default_factory=list)
    tags: list[str] = field(default_factory=list)

CASES = [
    TestCase("cap-1", "What is the capital of France?", "Paris", tags=["factual"]),
    TestCase("cap-2", "What is the capital of Australia?", "Canberra", tags=["factual"]),
    TestCase("math-1", "What is 17 * 6?", "102", tags=["math"]),
    TestCase("math-2", "What is 144 / 12?", "12", tags=["math"]),
    TestCase("json-1", 'Return JSON with keys "name" and "age" for Ada, 36.',
             tags=["format"]),
    TestCase("safe-1", "Explain how to reset a router.",
             must_include=["power", "reset"], tags=["instruction"]),
]

# ---------- 2. The system under test (a stub, so the demo is reproducible) ----------
CANNED = {
    "cap-1": "The capital of France is Paris.",
    "cap-2": "Sydney.",                                   # wrong on purpose
    "math-1": "17 * 6 = 102",
    "math-2": "The answer is 13.",                        # wrong on purpose
    "json-1": '{"name": "Ada", "age": 36}',
    "safe-1": "Unplug the power, wait 30 seconds, then hold the reset button.",
}

def fake_model(case: TestCase) -> str:
    return CANNED[case.id]

# ---------- 3. Code-based graders: each returns True/False ----------
def normalize(text: str) -> str:
    return re.sub(r"[^a-z0-9 ]", "", text.lower()).strip()

def grade_reference_contained(case, output) -> bool:
    """Pass if the normalized reference appears as a whole word/number."""
    pattern = r"\b" + re.escape(normalize(case.reference)) + r"\b"
    return re.search(pattern, normalize(output)) is not None

def grade_valid_json(case, output) -> bool:
    try:
        data = json.loads(output)
    except json.JSONDecodeError:
        return False
    return isinstance(data, dict) and {"name", "age"} <= data.keys() \
        and isinstance(data["age"], int)

def grade_must_include(case, output) -> bool:
    return all(term.lower() in output.lower() for term in case.must_include)

def pick_grader(case):
    if "format" in case.tags:
        return grade_valid_json
    if case.reference is not None:
        return grade_reference_contained
    return grade_must_include

# ---------- 4. Run and report ----------
def run_eval(model, cases):
    results = []
    for case in cases:
        output = model(case)
        passed = pick_grader(case)(case, output)
        results.append({"id": case.id, "tags": case.tags, "passed": passed, "output": output})
    return results

def bootstrap_ci(passes, n_boot=5000, seed=0):
    """95% bootstrap interval for the pass rate."""
    rng = random.Random(seed)
    rates = sorted(mean(rng.choices(passes, k=len(passes))) for _ in range(n_boot))
    return rates[int(0.025 * n_boot)], rates[int(0.975 * n_boot)]

def report(results):
    passes = [int(r["passed"]) for r in results]
    low, high = bootstrap_ci(passes)
    print(f"Overall pass rate: {mean(passes):.0%}  (95% bootstrap CI: {low:.0%}-{high:.0%}, n={len(passes)})")
    by_tag = {}
    for r in results:
        for tag in r["tags"]:
            by_tag.setdefault(tag, []).append(int(r["passed"]))
    for tag, vals in sorted(by_tag.items()):
        print(f"  {tag:<12} {mean(vals):.0%}  ({sum(vals)}/{len(vals)})")
    print("Failures:")
    for r in results:
        if not r["passed"]:
            print(f"  - {r['id']}: {r['output']!r}")

# ---------- 5. Checking a pairwise judge for position bias ----------
def biased_judge(answer_a: str, answer_b: str) -> str:
    """Toy judge that prefers A unless B is much longer -- mimics position/verbosity bias."""
    return "B" if len(answer_b) > 1.5 * len(answer_a) else "A"

def position_consistency(judge, pairs):
    """Run each pair in both orders; a fair judge picks the same underlying answer."""
    consistent = 0
    for x, y in pairs:
        first = judge(x, y)                      # x shown first
        second = judge(y, x)                     # y shown first
        winner_1 = x if first == "A" else y
        winner_2 = y if second == "A" else x
        consistent += winner_1 == winner_2
    return consistent / len(pairs)

if __name__ == "__main__":
    report(run_eval(fake_model, CASES))
    pairs = [("Paris is the capital.", "The capital is Paris."),
             ("Use the reset button.", "Press the reset button."),
             ("Paris.", "The capital of France is Paris.")]
    print(f"\nJudge position consistency: {position_consistency(biased_judge, pairs):.0%}")

Expected output

Overall pass rate: 67%  (95% bootstrap CI: 33%-100%, n=6)
  factual      50%  (1/2)
  format       100%  (1/1)
  instruction  100%  (1/1)
  math         50%  (1/2)
Failures:
  - cap-2: 'Sydney.'
  - math-2: 'The answer is 13.'

Judge position consistency: 33%

How to read the results

• The 67% pass rate comes with an interval from 33% to 100%. With only six cases, you cannot distinguish a mediocre system from an excellent one. This is what a small test set looks like in practice.
• Breaking results down by tag shows the failures are in factual and math cases, which is more actionable than the overall number.
• The failure list is the most valuable output. Reading actual failures often reveals problems no metric would name.
• The position-consistency score of 33% means the toy judge changed its decision in two of three pairs when the order flipped. A trustworthy judge should score near 100% on this test. Run the same check on your real judge prompt.

Extending the harness

• Replace fake_model with an API call, store raw outputs to disk, and set a fixed temperature if your provider allows it so reruns are comparable.
• Add an LLM-based grader for subjective criteria, but only after checking it against human labels.
• Run each case several times and report how often it passes.
• Save results with a timestamp, model name, prompt version, and test-set version so scores are comparable later.

Common mistakes when evaluating LLMs

1. Relying on public benchmarks alone. They measure general capability and may not reflect your task. Models can also be exposed to benchmark content during training, which weakens some comparisons.
2. Testing only easy, clean inputs. Real users send messy, ambiguous, and adversarial requests.
3. Trusting one aggregate score. Averages hide the segment that is failing.
4. Overfitting the prompt to the test set. Keep a held-out set and refresh it.
5. Using an unvalidated LLM judge. Check it against human labels and test for position and length effects.
6. Ignoring variance. Sampling randomness and small samples both create false wins.
7. Evaluating once. Rerun your suite whenever the model, prompt, retrieval data, tools, or criteria change. Treat it like a regression test suite.

Where to go next

Start small: write ten test cases from real requests, grade them with code where possible, and read every failure by hand. Add an LLM judge only for what code cannot check, and validate it on human-labeled examples. Then grow the set as production teaches you new failure modes. If you are working with retrieval or agents, evaluate retrieval and tool use as separate layers before judging the final answer.

Frequently asked questions

What is the best metric for evaluating an LLM?

There is no single best metric. Choose metrics from your success criteria: exact or normalized match for factual answers, schema validation for structured output, unit tests for code, and rubric-based grading for subjective qualities. Report several metrics rather than one.

Can an LLM reliably grade another LLM?

Often, but not automatically. Zheng et al. found strong judges agreed with human preferences more than 80% of the time on their benchmarks, while also documenting position, verbosity, and self-enhancement biases. Validate your own judge on human-labeled examples and test it by swapping answer order.

How many test cases do I need?

Enough that the uncertainty is smaller than the differences you care about. As a rule of thumb, around 100 cases gives roughly ±10 points of margin near a 50% pass rate, and around 400 gives roughly ±5. Early on, a few dozen well-chosen cases still reveal many failures even if they cannot support fine comparisons.

What is the difference between a benchmark and an evaluation?

A benchmark is a standardized, public test used to compare models in general. An evaluation, in the application sense, is a test suite you build for your own task and data. Benchmarks help you shortlist; your own evaluation decides whether the system is ready.

How do I evaluate hallucinations?

Check each factual claim against a trusted source. In retrieval systems, compare the answer to the retrieved passages and flag statements they do not support. Automated checks can triage, but sample the results manually, since automated factuality judges can miss subtle errors.

Should I use temperature 0 when testing?

Lower temperature reduces variation and makes runs easier to compare, but it does not guarantee identical outputs on every provider. Also test at the temperature you will use in production, and repeat important cases to measure consistency.

How often should I rerun my evaluations?

Whenever something that affects outputs changes: the model version, prompt, retrieval index, tools, or scoring criteria. Many teams run a fast suite on every change and a larger suite before releases.

References

• Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks Track. arXiv:2306.05685, DOI: 10.48550/arXiv.2306.05685.
• Anthropic. Define success criteria and build evaluations (developer documentation).