What is an eval?

What an eval actually is

An eval has three parts: a dataset of inputs, a grader that decides whether an output is correct, and a runner that walks the two. The dataset is usually a JSONL file where each row carries the user input and whatever ground truth the grader needs. The grader can be a plain string match, a regex, a code check that parses the output, or another model prompted to judge the response against a rubric. The runner sends each input through the system under test, hands the output to the grader, and reports pass rate, per-row scores, latency and cost.

The word is used two ways. Providers use it for the framework: OpenAI's Evals API and Anthropic's evaluation tool both expose the same shape of dataset, grader, run. Researchers use it for a public benchmark like MMLU or GSM8K, which is a specific frozen dataset with a specific metric. The mechanics are the same; the difference is who owns the dataset and what the number is meant to say.

How is an eval different from a unit test?

A unit test asserts that a deterministic function returns a specific value. An eval assumes the system under test is not deterministic and reports a distribution instead. Two runs of the same eval on the same model version can produce different pass rates because sampling is stochastic, batched kernels reorder floating point sums, and providers roll silent updates to a model. So evals report aggregate metrics (pass at N, mean score, percentiles) and are usually run over dozens to thousands of rows, not the handful you would use for a unit test.

The other split is the grader. A unit test compares to a literal; an eval compares to a rubric. Simple tasks like classification, extraction and tool argument selection can use exact match or schema validation. Open-ended tasks fall back to a judge model, which is another LLM prompted to score the answer against criteria you write. LLM judges are cheap and flexible, but they inherit the biases of the judge, so you calibrate them against a small hand-labelled slice before you trust their numbers.

Where evals fit in a real workflow

An offline eval runs on a fixed dataset before you ship. It answers questions like: does this new prompt improve extraction accuracy on last quarter's tickets, does switching to a smaller model keep tool-call precision above 95 percent, does adding a retrieval step reduce hallucination on the FAQ set. Because the dataset is fixed, offline evals are what you wire into CI and what you diff against a baseline when reviewing a prompt change.

An online eval runs on production traffic after you ship. It scores a sample of real requests, usually with a judge model or a lightweight heuristic, and feeds the numbers to a dashboard so you can watch quality drift as usage shifts. The two are complementary: offline evals catch regressions before they land, and online evals catch the ones your offline dataset did not cover. Providers now bundle both. OpenAI ships an Evals API for offline runs and traces for online scoring, and Anthropic documents the same split under its develop-tests guide.

Public benchmarks sit alongside these. Suites like MMLU, HumanEval and the EleutherAI lm-evaluation-harness are useful for choosing a base model, but a leaderboard score does not tell you how the model behaves on your prompt, your tools or your users. Treat them as a first filter, then build your own eval for the task that actually matters.

The mistakes that make eval numbers meaningless

The first is dataset leakage. If your eval rows came out of the same production logs you used to write the prompt, the score reflects memorisation rather than generalisation. Hold out a slice from the moment you build the set and never train, tune or prompt-engineer against it.

The second is grading the wrong thing. Exact string match is unforgiving on tasks where two answers can be equally right, and a judge model that only sees the answer cannot tell whether the reasoning was correct or the citation was real. Match the grader to the failure mode you care about, and where the failure mode is fuzzy, log the raw outputs so a human can spot-check the judge's calls.

The third is running too few rows. A 20-row eval swings by 5 percentage points on noise alone, so a prompt tweak that moves the number by 3 points is telling you nothing. Anthropic's guidance recommends hundreds of cases where you can afford them, and reporting confidence intervals rather than a single pass rate.

# dataset.jsonl
# {"input": "ship it Tuesday at 3", "expected": {"day": "Tue", "hour": 15}}
# {"input": "push to prod tomorrow noon", "expected": {"day": "+1", "hour": 12}}
# ... 300 more rows ...

import json, anthropic
client = anthropic.Anthropic()

def run_one(row):
    resp = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=200,
        temperature=0,
        system="Extract {day, hour} as JSON. Nothing else.",
        messages=[{"role": "user", "content": row["input"]}],
    )
    try:
        got = json.loads(resp.content[0].text)
    except json.JSONDecodeError:
        return {"pass": False, "reason": "not-json", "raw": resp.content[0].text}
    ok = got == row["expected"]
    return {"pass": ok, "got": got, "expected": row["expected"]}

rows = [json.loads(l) for l in open("dataset.jsonl")]
results = [run_one(r) for r in rows]
passed = sum(1 for r in results if r["pass"])
print(f"pass rate: {passed}/{len(rows)} = {passed/len(rows):.1%}")

# Persist per-row results so a later prompt diff can be attributed
# to specific failing rows, not just to a moving aggregate.
json.dump(results, open("run-2026-08-30.json", "w"), indent=2)