Evals for an MCP server or a CLI
Unit tests check your code. Evals check whether a real agent can still do the job through it. What to test, and why a judge is needed.
Your unit tests pass. Your server returns the right values. And an agent using it still cannot complete the user's task, because it read a tool description differently after a model update.
That failure is invisible to every test you have.
What an eval is
An eval runs a real agent against your MCP server or CLI on a real user workflow, and scores whether the user's task actually succeeded. A unit test checks that a tool returns the right value. An eval checks that an agent can still do the job using it.
The difference is the unit. A unit test asserts on a value. An eval asserts on an outcome after a sequence of decisions you did not control.
Why the failures live outside your code
Almost everything that breaks an agent integration is something a normal test cannot reach.
A tool description that reads differently to a new model. You did not change it. The model changed. The agent now reaches for the wrong tool in a case it used to get right.
An argument format the agent stops guessing correctly. It used to send an ISO date. Now it sends "last Tuesday", because your description does not say and the model's default shifted.
A tool that is now never called. Your description competes with the others for the agent's attention, and something you added made it lose.
A sequence that no longer completes. Every call succeeds. The order changed and the outcome is wrong.
Context pressure. You added five tools. The menu got long enough that the agent stops reading it properly.
None of those produce an error. Your logs look healthy. See MCP analytics for how they look in production.
What to put in a suite
Start from what users actually do, not from your tool list.
Real workflows, not single calls. "Refund the last duplicate charge for this customer" rather than "call create_refund". The multi-step ones are where the failures are.
The ambiguous cases. A request that could reasonably map to two tools. That is where a description does its work, and where a model update shows up first.
The failure paths. A missing record, a permission error, an invalid argument. An agent that handles a failure badly loses the user's task just as thoroughly as one that fails outright.
One case per bug you have already had. The cheapest suite is the one built from real incidents. If a session failed in production once, it belongs in the suite.
The boring happy path. It breaks more often than you would expect.
Why it needs a judge
The output of an eval is a session: a transcript, a set of calls, some side effects. Something has to read that and decide whether the user's task was done.
At any volume that something is a model. See judge model. Two rules carry most of the value:
Use a model family that does not build the agent you are testing. A model recognises its own family's prose, which is a bias you cannot detect afterwards and can trivially avoid.
Write the rules down. With explicit rules, three copies of our judge agreed with each other 97.9% of the time. Without them, 84%. Thirteen points, from writing down what counts.
Flakiness is the shape of the problem, not a defect
This is the part teams find hardest coming from unit tests.
The same agent, on the same input, does not always do the same thing. In our experiments the same agent on the same repository disagreed with itself about a quarter of the time. A single eval run is a coin flip with an opinion.
Design for it:
- Several runs per case. Three at minimum, five if the case matters.
- A threshold, not a binary. "Passes four of five" is a result. "Passed" is not.
- Pin the agent version. These tools change, and results from two versions are not comparable. Record the version with every run.
- Fail loudly on a shape you do not recognise. When an agent's output format changes, quarantine that version rather than parsing it wrongly. We do this and it has cost us hours of throughput and never let a wrong result through.
- Never retry a failure into a pass. A run that failed is a result.
When to run them
On every deploy, for the small suite of workflows that must not break.
Nightly, for the full suite, because the thing that breaks you is usually not your deploy. It is a model or client update you did not make.
That second one is the argument for having evals at all. Your code did not change and your product stopped working, and without a nightly run you find out from a user.
What to do with a failure
Read the session before touching the code. In our experience the fix is a tool description more often than it is an implementation, and rewriting a description is a ten-minute change that a normal debugging session would never have arrived at.
Where this sits
Three related things worth keeping apart:
| Question | Measured by | |
|---|---|---|
| Agent discoverability | Does an agent choose us at all? | Install share |
| MCP analytics | What do real users do, and where do they fail? | Sessions from production |
| Evals | Does it still work after this change? | Replayed workflows, judged |
Analytics tells you what to write an eval for. The eval stops that failure coming back.
Armature ships both: session capture with redaction before storage, and eval suites that replay real workflows against an MCP server or a CLI on every deploy and nightly, with a judge on every run. Suites can be drafted from real analytics sessions, which is the shortest path from "a user hit this" to "this cannot happen again".
Common questions
What is an eval for an MCP server?
A test that runs a real agent against your server on a real user workflow and scores whether the user's task succeeded. Unit tests check that a tool returns the right value; an eval checks that an agent can still complete the job using it.
Why not just use unit tests?
Because the failures are not in your code. A tool can return correct values while an agent misreads its description, picks the wrong tool, or formats an argument differently after a model update. Only running the agent catches that.
Why does an eval need a judge?
Because the output is a session, not a value. Something has to read the transcript and decide whether the user's task was actually done, and at any volume that something is a model.
How do you stop evals being flaky?
Accept that they are stochastic and design for it: several runs per case, a pass threshold rather than a binary, pinned agent versions, and a judge with explicit rules. Three copies of our judge agreed with each other 97.9% of the time with rules and 84% without.
Where this comes from
Armature ran 5,292 judged sessions with Claude Code, Codex and Cursor inside 51 realistic codebases, and published every run. The numbers on this page come from that work.