How to Actually Evaluate an LLM in Production

Every model selection meeting I’ve sat in over the past year has the same moment. Someone shares their screen, pulls up a public leaderboard, and the room nods at a table of numbers nobody can map to the product. The model at the top wins the meeting. Whether it wins in production is a different question, and usually nobody is assigned to answer it.

The reflex is understandable. Benchmarks are public, comparable, and free. Building your own evaluation takes work, so teams borrow someone else’s. Then the model ships, a prompt gets improved on a Friday afternoon, quality drifts quietly for two weeks, and the first real evaluation happens inside a customer complaint.

My narrower point is this: evaluating an LLM in production isn’t a research problem. It’s an engineering discipline, closer to regression testing and observability than to benchmarking. You need a golden set built from your own traffic, a judging method you’ve calibrated, a gate that runs on every change, and online signals for everything the offline suite can’t see. That’s the whole pipeline, and this post walks through each piece. It won’t tell you which frontier model is best, because for your specific workload that question has no general answer.

Key takeaways

• Public benchmarks sort models into rough capability tiers. They don’t predict success on your task, because your production traffic is a different distribution from any benchmark.
• The highest-value evaluation asset you can own is a golden set: a few hundred real production inputs with agreed-correct outputs, versioned like code.
• LLM-as-judge is usable, but only after you’ve measured its agreement with human labels on your own data. An uncalibrated judge is a random number generator with confidence.
• Treat every prompt or model change like a database migration: nothing ships without the eval suite passing, and the score diff gets reviewed, not vibed.
• Offline evals catch regressions you anticipated. Online signals, retries, escalations, malformed outputs, catch the failures your golden set never imagined.

Why don’t public benchmarks predict production behaviour?

Because benchmarks measure general capability on curated academic tasks, and your product is neither general nor curated. A model’s score on a multiple-choice knowledge test tells you it belongs to a certain capability tier. It tells you almost nothing about whether it will extract the right fields from your invoices, follow your tone guidelines, or refuse the requests you need it to refuse.

The reflex here is to treat the leaderboard as a shopping list: sort descending, pick the top affordable option, done. Three things break that logic. First, distribution mismatch. Benchmark questions look nothing like your traffic, and models that are close on a benchmark can be far apart on a narrow task. Second, saturation. When every serious model scores in the same band on a popular benchmark, the ranking within that band is mostly noise. Third, contamination. Benchmark data leaks into training sets, so scores drift upward in ways that say more about data pipelines than about capability.

Would you pick a database engine off a synthetic benchmark result alone, without running your own workload against it? Most senior engineers wouldn’t. LLMs deserve the same skepticism, arguably more, because their failure modes are quieter.

 [PERSONAL EXPERIENCE] Two same-tier models diverging wildly on a narrow extraction task despite near-identical benchmark scores.

I’ve watched two models from the same capability tier, separated by a couple of points on the public leaderboards, differ enormously on a structured extraction task. One followed the output schema nearly every time. The other invented fields under pressure. Nothing in any public number predicted that. Twenty minutes with a task-specific eval set did.

Where’s the line? Benchmarks earn their keep at the shortlisting stage. They’ll reliably tell you that a small local model isn’t in the same class as a frontier one, and they’ll prune a list of twenty candidates down to four or five worth testing. Once you’re comparing models within the same tier for your specific task, the leaderboard has nothing left to offer. Your own eval takes over from there.

 The real cost of an LLM call → blog 3, The Real Cost of an LLM Call vs a SQL Query.

What should you actually measure?

Measure task success, defined by your product, on your own traffic. Not “model quality” in the abstract, not a one-to-ten helpfulness score, but the concrete question your feature exists to answer: did the summary keep every factual claim, did the extraction match the schema, did the reply resolve the ticket.

The asset that makes this possible is a golden set: a sample of real production inputs, each paired with an agreed-correct output or a grading rubric, versioned in the repo like any other test fixture. Building one is unglamorous work. Pull a few hundred requests from production logs, stratify them across the intents you actually see, sit down with whoever owns the product and label them. Most teams find that 200 to 500 well-chosen examples give a stable signal. Below roughly a hundred, run-to-run variance swamps the differences you’re trying to detect.

Where do you store the results? Anywhere queryable. A plain Postgres table is enough, and it keeps the history honest:

CREATE TABLE eval_runs (
  run_id        uuid PRIMARY KEY,
  git_sha       text NOT NULL,        -- prompt + code version
  model_id      text NOT NULL,
  golden_set    text NOT NULL,        -- versioned dataset name
  case_id       text NOT NULL,
  passed        boolean,
  score         numeric(4,3),
  failure_tag   text,                 -- schema_violation, hallucination, refusal...
  created_at    timestamptz DEFAULT now()
);

That failure_tag column earns its place quickly. Aggregate pass rates tell you something moved. Failure tags tell you what moved, and they’re what turns an eval run into a debugging session instead of an argument.

One habit worth stealing from test engineering: define pass criteria per task type. Structured outputs get checked mechanically against a schema. Deterministic answers get exact or normalized matching. Only the genuinely open-ended cases get a rubric, because rubric grading is the most expensive and least reliable tier of the three.

Where’s the line? Golden sets go stale. If your product’s traffic shifts, new intents, new document formats, a new user segment, the set stops representing reality and pass rates become a comforting fiction. Refresh it on a cadence, monthly is a sane default, and treat any big product change as a trigger to re-sample. And if the product is changing faster than you can label, accept that offline evals will lag and lean harder on the online signals covered below.

Can you trust an LLM to judge another LLM?

Yes, within limits you’ve measured. No, if you haven’t measured them. LLM-as-judge is the only affordable way to grade open-ended outputs at scale, and it’s also the easiest place in the whole pipeline to fool yourself.

The reflex is to write a judging prompt (“rate this answer one to ten for helpfulness”), point a strong model at it, and treat the output as ground truth. What that produces is a number with excellent uptime and unknown meaning. Before a judge grades anything that matters, calibrate it: take a slice of your golden set that humans have already labeled, run the judge over it, and measure agreement. If the judge agrees with your human labels at a rate you’d accept from a new team member doing review work, it’s earned a seat. If not, the judging prompt needs work before the pipeline does.

Known biases are worth designing around rather than discovering. Judges tend to prefer longer answers regardless of quality. In pairwise comparisons they favour whichever answer appears in a given position, so swap the order and average. And models show some preference for text that resembles their own output style, which argues for using a judge from a different family than the model being judged when the comparison is close.

 [UNIQUE INSIGHT] Rubric decomposition: several binary checks beat one holistic score.

The single change that most improves judge reliability, in my experience, is decomposition. Don’t ask for one holistic score. Ask five binary questions: does the answer address the question, does it contradict the source, does it follow the format, does it include unsupported claims, does it stay within scope. Binary checks agree with humans far more often than ten-point scales do, and the disagreements are easier to audit.

Where’s the line? Judges fail where they lack the knowledge to grade. Legal reasoning, medical claims, and subtle code correctness are places where a judge model confidently blesses wrong answers. For code, execute it; an interpreter is a better judge than any model. For expert domains, keep a human sampling loop no matter how good the automated agreement looks. And never let the same prompt that generated an answer also grade it.

 Retrieval quality is a search problem → blog 4, Why Your RAG System Is Actually a Search Problem.

How do you stop a prompt change from silently breaking production?

Run the eval suite on every change, the same way you run tests on every commit. A prompt edit is a deploy. It can change behaviour as much as swapping the model can, and in most teams it happens ten times as often, with a tenth of the ceremony.

That means prompts live in version control, not in a dashboard text box. It means the CI pipeline runs the golden set against any change to prompts, retrieval configuration, or model version, and posts the score diff on the pull request. And it means someone reads that diff. A two-point drop in overall pass rate might hide a twenty-point drop on one intent that a stratified view would surface instantly.

The shape of the gate is ordinary test tooling:

def test_extraction_suite(golden_set, model_under_test):
    results = run_eval(golden_set, model_under_test)
    assert results.pass_rate >= BASELINE - TOLERANCE
    # fail loudly on any regression in critical tags
    assert results.failures_by_tag["schema_violation"] == 0

If this looks like the discipline database teams apply to schema migrations, that’s the point. The move that makes migrations safe isn’t a clever tool, it’s the refusal to ship unreviewed structural change. Prompts deserve the same refusal. For bigger changes, model swaps especially, add a shadow phase: run the candidate on a copy of live traffic, compare outputs against the incumbent offline, and only then route real users.

Isn’t this expensive? Less than it looks. A few hundred eval calls per change is usually a rounding error next to production volume, and you can run the cheap mechanical checks on every commit while reserving judge-graded runs for merges.

Where’s the line? The suite can’t cover everything, and pretending otherwise breeds false confidence. If a full run costs too much to gate every commit, sample it, but never sample the mechanical checks; schema validation is nearly free. And a gate is only as good as its baseline: if pass rates have quietly eroded across ten small approved drops, the gate has been ratifying decline. Track the trend line, not just the diff.

 Sequencing discipline for risky changes → blog 9, Zero-Downtime Database Migrations: Patterns and Pitfalls.

What do you watch after the model is live?

Watch the signals users generate when the model fails them. Offline evals grade the failures you imagined. Production supplies the ones you didn’t, and users flag them constantly, just not through the feedback button.

The high-value signals are behavioural. Immediate retries and rephrasings of the same question mean the first answer missed. Escalations to a human channel right after a model interaction mean the model didn’t resolve the task. Heavy user edits to a generated draft are a direct, free quality measurement. On the system side, schema validation failures, tool-call errors, truncated outputs, and refusal rates are all cheap to count and brutally honest. Explicit thumbs ratings, by contrast, are sparse and skewed; treat them as a weak signal, not a metric to optimise.

 [PERSONAL EXPERIENCE] Retry-rate segmentation surfacing a failing intent that offline evals missed.

On one support automation rollout, the aggregate numbers looked fine for weeks while one intent quietly failed. What exposed it wasn’t any eval run. It was segmenting retry rate by intent, which showed users asking one category of question twice as often as any other before giving up. The golden set had three examples of that intent. Production had thousands.

That gap points at the practice that ties this whole post together: the flywheel. Every production failure you catch is a labeled example you didn’t have to invent. Route flagged interactions into a review queue, label a sample each week, and fold the confirmed failures into the golden set. Six months of this and your eval suite describes your actual failure surface instead of your launch-week guesses. It’s the least glamorous loop in AI engineering and the one that compounds hardest.

Where’s the line? Online signals are lagging and noisy. They tell you something broke after users have already felt it, which is exactly why they can’t replace the pre-deploy gate. And behavioural metrics can mislead in isolation: retry rates rise when a feature gets popular with new users, not only when quality drops. Read them against traffic mix, and confirm with sampled human review before declaring an incident.

So how do you actually decide ?

Four questions turn all of the above into a Monday-morning plan.

1. Can the output be checked mechanically? Schema validation, exact match, and code execution are the cheapest and most reliable graders you’ll ever have. Push as much of the eval surface as possible into this tier before reaching for a judge model.

2. Do you have real traffic yet? If not, write synthetic cases from the spec and accept that they’re placeholders. Plan to replace them with sampled production data within the first weeks after launch. A golden set built entirely from imagination measures your imagination.

3. How expensive is a bad output? This sets the human involvement floor. Low-stakes drafting can live on automated judges and behavioural signals. Anything touching money, health, or legal exposure keeps a permanent human sampling loop, however good the automated agreement gets.

4. How often do prompts and models change? Change frequency dictates automation. A prompt that changes weekly needs a fully automated CI gate. A model reviewed quarterly can afford a heavier, partly manual bake-off.

If your answers are “mostly mechanical, real traffic, low stakes, frequent changes,” you can run a lean, almost fully automated pipeline. The further your answers drift toward open-ended, synthetic, high-stakes, and infrequent, the more human judgment stays in the loop.

Evaluation is a habit, not a leaderboard

The practical default is unexciting: a golden set sampled from your own traffic, mechanical checks wherever possible, a calibrated judge for the rest, a CI gate on every prompt and model change, and a weekly loop that turns production failures into new eval cases. None of it requires a platform purchase. Most of it is the testing discipline your team already applies to ordinary code, aimed at a component that happens to be probabilistic.

The cautionary version is just as plain. Teams that skip this pick models off leaderboards, edit prompts in dashboards, and find out about regressions from customers. The system degrades in a way no dashboard shows, because nobody built the dashboard that would show it. By the time the complaint arrives, nobody can even say which of the last eleven prompt tweaks caused it.

So resist the pull of someone else’s numbers. Sample your traffic, label it, gate your changes, and let production teach your eval suite what failure looks like. That’s how you actually evaluate an LLM in production: not by reading scores somebody else generated, but by generating your own.

Frequently asked questions

• What is a golden dataset for LLM evaluation?

A golden dataset is a versioned collection of real production inputs paired with agreed-correct outputs or grading rubrics. Teams run it against every prompt or model change to catch regressions before deployment. A few hundred well-labeled examples sampled from live traffic beat thousands of synthetic ones written from a spec.

• How many examples do you need to evaluate an LLM in production?

Most teams get a stable signal from 200 to 500 labeled examples, provided they’re sampled from real traffic and stratified across the intents that actually occur. Below roughly a hundred, score changes between runs are mostly noise. Coverage of edge cases and labeling quality matter more than raw example count.

• Is LLM-as-a-judge reliable?

It can be, but only after calibration. Measure the judge’s agreement with human labels on your own data before trusting its scores, and design around position bias and verbosity bias. Decomposing grading into several binary checks improves reliability. An uncalibrated judge produces confident numbers that may not track what your users consider good.

• How often should you re-run LLM evals?

Run the offline suite on every prompt, model, or retrieval change, exactly like a test suite in continuous integration. Separately, refresh the golden set itself on a regular cadence, monthly is a reasonable default, and after any major product change, so the eval data keeps tracking how production traffic actually evolves.

• Do public benchmarks like MMLU still matter?

They matter for shortlisting. Benchmarks sort models into rough capability tiers and rule out options that are clearly too weak or too expensive for the job. Within a tier, they stop predicting anything useful about your specific task, and a golden set built from your own production traffic takes over.