Guide

How to measure whether an AI agent works well: a set of reference cases, scoring the results and testing after every change

This guide is for managers who have an AI agent in a pilot or in day-to-day use and have to tell the management board whether it works. We show how to build a set of reference cases from your company’s history, which measures to choose and how to catch a decline that no one will report. At the end you get a template for a one-page report for the board.

Author: Kacper Włodarczyk, CEO of ALGORCOMPPublished: August 23, 2026Reading time: 11 min readAI / AI agentsFor: Mid-sized company
How to measure whether an AI agent works well: a set of reference cases, scoring the results and testing after every change

Why a successful demo does not prove that an agent works

A demo of an agent has a built-in flaw: the presenter chooses the questions. They do not do it out of bad faith. They simply remember which cases worked and show those.

The team sees ten correct answers and assumes it will always be like that. But day-to-day work brings incomplete cases, unusual ones and ones the agent should not answer on its own.

The 2025 MIT NANDA report shows that only a small share of AI pilots translate into financial results. The authors identify systems that do not learn from feedback as the main obstacle. Without measurement there is no feedback, and so no improvement either.

Measurement also helps defend a project that works. “People are happy” loses out to a cost spreadsheet. “On 120 real cases, the agent produces a ready-to-send result 78% of the time, up from 41% at the start” needs no defence (illustrative figures).

How to build a set of reference cases

A reference set (known in the industry as a “golden set”) is a list of input cases together with the correct result. The result is approved by the person who is responsible for the quality of this work day to day, e.g. the customer service manager – not by the IT department or the vendor.

You take the cases from the company’s history. For a customer service agent, these are closed tickets from the last quarter. For an agent that reads documents, they are documents that have already been processed manually. The rule is simple: if a case did not happen in reality, it does not go into the set.

The set should be deliberately uneven. The proportions and numbers below are our practical guidance for getting started, not a standard. The last group matters most: an agent that never says “I don’t know” is more dangerous than an agent that makes more mistakes but knows its limits.

  • About 60% typical cases, i.e. those that make up most of the work.
  • About 25% difficult but regular cases: unusual wording, missing data, two matters in one message.
  • About 15% cases where the correct response is to decline or to hand over to a person.
  • In our experience, about 50 cases are enough to start with for one narrow process, and 150–200 for a broader scope.
  • Every quarter, add the cases where the agent failed in day-to-day work.
How to measure whether an AI agent works well: a set of reference cases, scoring the results and testing after every change

Which measures to choose for each type of agent

There is no single measure for all agents, because their mistakes cost different amounts. An agent that gives a customer an out-of-date delivery date does a different kind of damage from one that misreads a bank account number on a document.

For every type of agent, add one common measure: how often the agent correctly recognises that it should not act on its own. This measure translates best into the team’s trust, yet it rarely appears in vendors’ proposals.

Write down the measures and the acceptance threshold before the first measurement. The threshold is a business decision. For an assistant that suggests email content, 85% correct drafts may be enough. For an agent that writes data into the finance system, even 99% may be too little. The table below summarises the measures for four types of agent.

Quality measures by type of agent
Type of agentCore measuresWhat else to measure
Answers questionsAccuracy of answers; whether each answer can be traced to a source documentHow often it rightly declines when data is missing
Reads data from documentsAccuracy of each field; share of documents ready without correctionsAccuracy of critical fields separately, e.g. amounts and account numbers
Carries out operations in systemsShare of tasks completed correctly without human helpOperations carried out incorrectly with no sign of a problem
Prepares content and quotesShare of results accepted without reworkTime to a ready-to-send version

When the scoring can be done by a second AI model

Manually scoring two hundred cases after every change is unrealistic. That is why part of the scoring is handed to a second AI model, which compares the agent’s output with the reference and gives a score. This works provided you know how often this “judge” gets things wrong.

Checking this is simple. A person scores a random sample, e.g. 30–50 cases, and you compare their scores with the model’s. What matters is not just how often they agree, but also the direction of the errors. Research on scoring models (e.g. Zheng et al., 2023) shows that they tend to favour longer answers and are weaker at judging tasks that require calculation and reasoning.

Hence the division of work set out in the points below. Treat the scoring rules as a project document: every change to them changes all later results, so record which version was in force.

  • Check amounts, dates, document numbers and the presence of required fields with a simple program, not an AI model.
  • Leave to the scoring model what requires judging meaning: whether the content matches the reference, the tone, any caveats that were left out.
  • Use a different model for scoring from the one the agent uses.
  • Give it written scoring rules with examples of good and bad answers.
A manager compares an AI agent’s results with a set of reference cases in a table

The question “does the agent work?” has no answer until someone writes down what “works” means for fifty real cases and signs their name to it.

Testing after every change: where agents quietly break

The most underestimated risk is a change that does not come from you. The vendor releases a new model version, retires the old one, and the agent’s behaviour changes without anyone deciding it should. Often the change affects only some cases, so it is hard to notice.

Someone would report an outage. A decline of a few percentage points shows up differently. The team starts checking the results more carefully, then checks all of them, and after a quarter stops using the agent.

The solution is a regression test – a check that a change has not broken what worked before. The whole reference set runs automatically and is compared with the previous result. If the result falls below the agreed threshold, the change does not go into day-to-day use.

The list below says when to run the test and which two habits are worth adding. They cost little but help you avoid surprises.

  • Run the test after every change to the model’s instructions (the so-called prompt), the source data, the system connections and the model version.
  • Record the model version with every result, so that you can reconstruct what was measured.
  • When a vendor announces a new model version, test it before you are switched over to it. That way the change is your decision, not a surprise.

How to keep an eye on quality in day-to-day work

The reference set tests the agent on known cases. In day-to-day work, cases come up that are not in the set, so you also need a second, lightweight measurement carried out on an ongoing basis.

Three indicators collected automatically are enough: the share of cases closed without human involvement, the share handed over to a person, and the share of results a person corrected before use. The last one is the most valuable. Every correction is a ready-made case to add to the reference set.

On top of that, once a week someone reviews ten random cases. It is about a quarter of an hour’s work, and it catches problems the automated checks cannot see. The automated check compares the result with the reference; a person compares it with common sense.

A one-page report for the management board

The management board does not need technical details. It needs to know whether the agent works, what it delivers and whether anything went wrong. All of this fits on one page in the five points below.

The fourth point, the list of incidents, is more important than it seems. A board that sees problems reported and resolved trusts the figures more than a board that only sees successes.

If the agent talks to customers, also remember the law. From 2 August 2026, the AI Act (Article 50) requires an AI system to inform the person it is talking to that they are not talking to a human.

  • The result on the reference set compared with the previous quarter and with the starting point.
  • The number of cases handled by the agent in the period.
  • Working time saved, converted into hours and PLN, showing how it was calculated.
  • A list of incidents where the agent failed, and what was done about them.
  • Named people: who reviews the results and who decides to switch the agent off if quality falls below the threshold.

Related topics in the knowledge base

Read next

FAQ

Measuring AI agent quality – frequently asked questions

Questions that keep coming up when moving from a pilot to day-to-day use.

How much work does it take to build a reference set?
For one narrow process, usually a few days’ work by the person responsible for the process, with support from the implementation team. Most of the time goes on agreeing the correct answer where the company did not have a single one. This work pays off regardless of AI, because it brings order to the process itself.
Can the vendor build the reference set for us?
It can prepare the structure, the tools and the first version. But the correct answers must be approved by someone from your company. A set built entirely by the vendor measures what the vendor considers correct.
What result means the agent can go live?
There is no single threshold, because everything depends on the cost of an error. A practical method: have your staff handle the same set and compare the results. People make mistakes too, and the criterion “better than the current process and cheaper” is easier to defend than any percentage.
How often should we do a full review, apart from the tests after changes?
Once a quarter. During the review you add cases from day-to-day work to the set and check whether the measures still reflect what matters to the company.
Does the same apply to off-the-shelf tools such as Microsoft Copilot?
Partly. In that case you do not control the model version, but you can periodically test the tool on your typical tasks. It is particularly worth doing this after changes announced by the vendor.

About this page

About the author

Kacper Włodarczyk — CEO of ALGORCOMP

Kacper Włodarczyk

CEO of ALGORCOMP

Kacper specializes in implementing Microsoft 365 Copilot, Copilot Studio, Power Platform (Power Automate, Power Apps, SharePoint) and AI agents for mid-sized B2B companies in Poland. He leads dozens of projects covering AI strategy, Power Platform governance, document workflow automation and sales processes. In his writing he focuses on the practical side of AI adoption in organizations — from the first proof of concept to company-wide rollout — with particular attention to data security, compliance (GDPR, NIS2, the AI Act) and return on investment.

Meet the team

Want to find out how well your AI agent really works?

In a free consultation we will help you define quality measures for your agent and plan the first set of reference cases from your company’s history.

Featured

Related articles