The agent evaluation stack: what to measure before you ship
Accuracy on a golden set tells you almost nothing about an agent that takes actions. Here are the six layers we instrument on every build, and the thresholds we hold before a release goes live.
An agent that answers questions can be graded like a search system. An agent that files a ticket, moves a record, or writes to a customer cannot. Its output is not a string to compare against a reference; it is a sequence of decisions with side effects. Grading it on final-answer accuracy is like testing a surgeon on the diagnosis and ignoring the operation.
The reasons these systems get cancelled are measurement reasons. When Gartner predicted in June 2025 that more than 40 percent of agentic AI projects would be abandoned by the end of 2027, it named three causes: escalating costs, unclear business value and inadequate risk controls. Model quality was not among them. Each of the three is invisible without instrumentation — cost and value only exist if someone is counting them, and a risk control is only credible if you can show what the agent does when it is wrong.
Every agent we put into production carries six measurements, kept separate on purpose. When they are collapsed into one score, you lose the ability to tell which part broke.
The six layers
Retrieval. Did it find the right source material, measured independently of what it then said? Most answer failures we investigate turn out to be retrieval failures wearing a costume, and they are cheap to fix once separated.
Grounding. Is every claim traceable to something retrieved? An agent that is right by coincidence will be wrong by coincidence next month. We hold a hard floor here in regulated work, because grounding is what makes an answer defensible to an auditor.
Tool correctness. Did it call the right tool with the right arguments? This is ordinary integration testing, and it catches the failures that cost real money — the correct refund issued against the wrong account.
Trajectory. Did it get there in a sane number of steps? Loops, repeated retries, and creative detours are early warnings that the task decomposition is wrong, and they show up in latency and cost long before they show up in output quality.
Refusal calibration. Does it decline when it should, and only when it should? Both directions are defects. An agent that never refuses is a liability; an agent that refuses routine work quietly destroys the adoption case.
Regression. Does this release do everything the last one did? Every incident becomes a permanent test case. This is the layer teams skip, and it is the one that stops a model or prompt change from silently undoing three months of work.
One aggregate score tells you the agent got worse. Six tell you what to fix.
Thresholds, not dashboards
A measurement with no threshold is decoration. Before the first release we agree, in writing, the number on each layer below which the release does not ship and above which it does — and who is allowed to override it. The numbers differ by use case: an internal drafting assistant and a customer-facing entitlement agent do not deserve the same floor.
That agreement is worth more than the evaluation harness itself. It converts a subjective argument at the end of a sprint into a decision made calmly at the start, and it gives the operations team the thing they always ask for: a defined point at which they stop trusting the system.
Start smaller than you think
Fifty well-chosen cases drawn from real traffic beat two thousand synthetic ones. Take the transcripts of the work the agent is replacing, sample across the boring middle and the awkward tail, and have the person who currently does the job label them. The exercise usually reveals that the process itself was never as consistent as anyone claimed, which is useful information whether or not you ship the agent.
- Gartner, Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027, press release, 25 June 2025. gartner.com
- The six layers and thresholds described here are drawn from Biruni Applied AI engagements, not from published research.
- —Measure retrieval, grounding, tool correctness, trajectory, refusal calibration, and regression separately.
- —Agree the pass threshold per layer in writing before the first release, and name who can override it.
- —Every incident becomes a permanent regression case.
- —Fifty real labelled cases beat two thousand synthetic ones.