How to prove an AI agent works – golden sets, LLM-as-judge and regression tests

A demonstration has a built-in methodological flaw: the presenter selects the questions. Not out of bad faith – they simply remember which cases worked and show those. The team sees ten accurate answers and draws a conclusion about overall effectiveness, having in fact seen a sample selected in hindsight.

The scale of the phenomenon is documented. MIT Project NANDA's 2025 report found that 95% of generative AI pilots produced no measurable profit-and-loss impact, while tool testing was very widespread – over 80% of organizations had launched something and around 5% extracted value. The authors called that gap the divide between adoption and transformation.

From an implementation standpoint the conclusion is practical: what separates those two groups is neither a better model nor a bigger budget, but the existence of a feedback loop. A team that measures, improves. A team that does not measure cannot say after a few months whether things are better than at the start – and then the budget disappears at the next review.

There is a second, quieter reason to measure: without a number you cannot defend a project that genuinely works. A manager saying „people are happy” loses to a spreadsheet of savings. A manager saying „on a set of 120 real cases the agent produces a send-ready result in 78% of them, up from 41% at the start” does not have to defend anything.

A reference set is a list of input cases together with the correct outcome, approved by the person accountable for the process. Not by the IT team and not by the vendor – by whoever answers day to day for the quality of that work.

The material comes from history. For a customer service agent, that is closed tickets from the last quarter. For a document agent – documents processed manually, together with what was extracted from them. For a sales agent – enquiries somebody has already answered. The rule is simple: if the case did not occur in reality, it does not belong in the set.

The mix should be deliberately uneven. Around 60% typical cases – the ones making up most of the volume. Around 25% hard but regularly occurring: unusual phrasing, incomplete data, two matters in one ticket. And around 15% where the correct answer is refusal or handover to a human. That last group is most often omitted and most diagnostic – an agent that never says „I do not know” is more dangerous than one with a lower score.

On size: 50 cases suffice to start for one narrow process; 150–200 give stable results for an agent covering a broader scope. Above that, the cost of maintaining the set rises while the information gain falls. The set is alive – each quarter you add cases where the agent failed in production, because those are precisely the situations you were unable to anticipate.

There is no single universal effectiveness measure, because different agents make differently expensive mistakes. An agent that quotes an outdated delivery date in customer service causes a different problem from an agent that reads the wrong account number in a document process.

For a question-answering agent you measure factual accuracy, grounding in a source (whether the answer can be pointed to in a document) and the share of correct refusals. For a document-processing agent – extraction accuracy per field, the share of documents ready without correction, and separately the accuracy on critical fields where an error costs most. For an agent that performs operations – the share of tasks completed correctly without intervention and, most importantly, the share of operations wrongly executed with no signal that anything went wrong.

To each type, add one shared measure: the share of cases where the agent correctly recognized it should not act alone. It translates best into team trust and almost never appears in vendor proposals.

Set the metrics before the first measurement and record them together with an acceptance threshold. The threshold is a business decision, not a technical one: for an assistant suggesting email copy, 85% may be entirely sufficient; for an agent writing to a financial system, 99% is sometimes not enough.

Manually scoring two hundred cases after every change is unrealistic, so part of the scoring goes to a second model that compares the agent's answer with a reference and issues a verdict. That works – provided you know how wrong the judge tends to be.

Calibration means a human scores a random subset, say forty cases, and you compare their scores with the judge's. What interests you is not only overall agreement but which way the judge leans. The typical biases are well documented: judge models more readily accept longer and more confidently phrased answers, handle numbers and dates less well, and tend to score texts stylistically similar to their own more leniently.

Hence a practical division of labour. Anything checkable deterministically – amounts, dates, document numbers, identifiers, presence of a required field – check with code, not with a model. Leave the judge what requires assessing meaning: whether the answer is substantively consistent with the reference, whether the tone fits, whether an important caveat was omitted.

It also helps for the judge to be a different model from the one being evaluated, and to score against an explicit, written rubric with examples of good and bad ratings. The judge's rubric is a project artefact just like the reference set – it is versioned, because changing it changes every result retroactively.

The most underrated risk in running an agent is a change that does not come from you. The model vendor releases a new version, the old one is retired, somebody flips a configuration – and the agent's behaviour changes in a way nobody planned. The change is often small and uneven: 95% of cases behave identically and 5% start coming out differently.

An outage would be easier. Somebody reports an outage. A few percentage points of degradation shows up as the team quietly starting to check results more carefully, then checking all of them, and after a quarter no longer using the tool. Nobody files a ticket, because there is no single event to file.

A regression test solves this with one mechanism: the whole reference set runs automatically after every change to prompts, source data, tool definitions and model version, compared against the previous result, with an alert threshold on any drop. A drop below the threshold blocks the change from shipping – exactly as a failing test blocks code.

Add two habits that cost little and save projects. First: record the model version alongside every result, so you can reconstruct what you measured on. Second: when the vendor announces a version change, run the set on the new version before you are switched over – that turns migration into a decision rather than a surprise.

A reference set measures quality under laboratory conditions. Production brings questions the set does not contain, so a second measurement layer is needed – light but continuous.

The minimum is three automatically collected indicators: the share of cases closed without human involvement, the escalation rate, and the share of cases where a human corrected the agent's output before using it. That last one is the most valuable, because corrections are ready-made material for extending the reference set – the cheapest source of knowledge about where the agent actually fails.

Add a weekly spot check: ten random cases scored by a human. Ten cases is about a quarter of an hour a week, and it catches drift the automation cannot see, because the automation compares against a reference rather than against common sense.

The board report fits on one page and has four items: performance on the reference set compared with the previous quarter and with the starting point, the volume handled by the agent, time saved converted into headcount or currency, and a list of incidents together with the response. The last item matters more than it seems – a board that sees incidents reported and handled trusts the numbers more than a board shown nothing but successes.

It is worth adding one sentence on accountability: who reviews the results and who decides to withdraw the agent should quality fall below the threshold. According to Capgemini Research Institute research, only 48% of organizations have a written split of responsibility between employee and agent – and it is a document whose absence surfaces precisely when it is most needed.