How to get picked for evals by coding agents
Langfuse took 31% of 478 judged evals sessions. What the numbers say a vendor in this category should do.
If you sell eval tools, this page is the part of the market no dashboard shows you: what a coding agent does when a developer asks for evals and never compares vendors.
The numbers come from 478 judged sessions with Claude Code, Codex, Cursor, Grok Build CLI and Muse Code, spread across 2 realistic codebases, with every session read by a judge.
Read the spread before the totals. 478 sessions is a lot of runs, and 2 codebases are not a lot of variety. With two codebases, a result is as much about those two projects as about the category. The panel had 2 with the evals seam open, and running on repositories where the request does not make sense would have been worse than running on fewer. Weigh what follows accordingly.
What coding agents choose for evals
Across 478 judged sessions, Langfuse was chosen most often, in 31% of runs. Promptfoo was second with 15%. In 34% of runs the agent wrote the code itself and adopted no product at all.
| # | Product | Runs won | Share |
|---|---|---|---|
| 1 | Built in-house (no product adopted) | 161 | 34% |
| 2 | Langfuse | 148 | 31% |
| 3 | Promptfoo | 70 | 15% |
| 4 | Arize Phoenix | 32 | 7% |
| 5 | Braintrust | 23 | 5% |
| 6 | Inspect AI | 16 | 3% |
| 7 | Helicone | 12 | 3% |
| 8 | LangSmith | 9 | 2% |
| 9 | pydantic-evals | 4 | 1% |
| 10 | DeepEval | 1 | 0% |
Full board, every run replayable: the evals leaderboard.
What the shape of this category means
The leader takes only 31% of runs. This category is genuinely open and the ordering can be moved.
With the top product at 31%, evals is decided in the moment, from what the agent reads and what it finds in the repository. Nothing is locked in, which is the best situation a vendor can be in and the one where the work pays fastest.
The order here is set by the quality of what an agent can read and by whether your product is already present in the codebase. Both are things you can change.
The agents do not agree with each other
In this category Claude Code, Codex and Cursor all put Langfuse first, which is less common than it sounds: across the 31 categories we measured, Claude Code and Codex disagreed on the leader in 16 of them.
Grok Build CLI (22 sessions) and Muse Code (72 sessions) join this table with fewer runs, so read their rows as indicative.
| Agent | Runs | Picked most often |
|---|---|---|
| Claude Code | 120 | Langfuse (46) |
| Codex | 144 | Langfuse (48) |
| Cursor | 120 | Langfuse (40) |
| Grok Build CLI | 22 | Langfuse (6) |
| Muse Code | 72 | Langfuse (8) |
Even where they agree, they get there differently. In this category Codex ran a web search in 99% of its sessions and Claude Code in 37%, so what you publish reaches nearly all of Codex's evals sessions and more than a third of Claude Code's.
Who is asking changes the answer
Every request in this experiment was written as a specific kind of person. In this category the leader changes with the person.
| Who is asking | Runs | Picked most often |
|---|---|---|
| Junior developer | 241 | Langfuse |
| Senior engineer | 237 | Promptfoo |
That is 2 different products winning evals for 2 kinds of buyer, out of the same 478 sessions. Nobody here is winning evals. They are each winning one kind of buyer.
If you sell to more than one of them, you need pages for each. See how to win the enterprise persona.
What you are really competing against
In 34% of runs, the agent wrote the code itself. That makes hand-written code the largest single competitor in this category, larger than most vendors in the table above.
This changes the job of your content. Before you argue that you are better than another vendor, you have to argue that the problem is harder than it looks. What breaks at volume. Which edge cases cost a weekend. What the maintenance actually costs after six months.
That argument has to exist as a page an agent can read, with specifics and numbers. "It is harder than you think" is not an argument. "Here are the four failure modes and what each one costs" is.
Considered, and never chosen
Because the judge records every product an agent raised and not only the one it picked, this board also shows who kept reaching the shortlist and losing. In evals the clearest case is Ragas: on the table in 42 sessions, chosen in none.
| Product | Raised in | Chosen in |
|---|---|---|
| Ragas | 42 sessions | 0 |
| Traceloop | 26 sessions | 0 |
Being rejected is a better position than being unknown, and a cheaper one to fix. The product is already in the agent's head and on the list. Whatever ended those 68 sessions is recorded in each transcript, one reason at a time.
What to do about it in evals
- Argue that the problem is harder than it looks, before you argue that you are better than a rival. 34% of evals sessions ended in hand-written code, so in roughly one session in 3 no vendor was in the running at all. Write the failure modes and the year-two maintenance cost, with numbers, in the documentation rather than the blog.
- Aim at second place first. Langfuse holds 31% and Promptfoo holds 15%. The gap between the default and the field is where the reachable sessions are.
- Report install share per persona, not as one number. In evals the leader changes with who is asking, so a blended figure averages markets that behave differently.
The work that applies to every category rather than to this one is written up separately: audit your documentation, write a quickstart an agent can follow, and how to measure install share.
Every eval tool on this board
One page per product, with its install share, the per-agent split, and how often it was raised without being chosen.
- Do coding agents recommend Langfuse? — chosen in 31% of sessions
- Do coding agents recommend Promptfoo? — chosen in 15% of sessions
- Do coding agents recommend Arize Phoenix? — chosen in 7% of sessions
- Do coding agents recommend Braintrust? — chosen in 5% of sessions
- Do coding agents recommend Inspect AI? — chosen in 3% of sessions
- Do coding agents recommend Helicone? — chosen in 3% of sessions
- Do coding agents recommend LangSmith? — chosen in 2% of sessions
Where these numbers come from
478 judged sessions in evals across 2 codebases, part of a published set of 15,287. Real coding agents at pinned versions, in sandboxes, inside realistic codebases, with a simulated project owner in the loop and a blind judge on every session. The full method is on one page: how we measured this.
Every evals run can be replayed on the board.
<!-- generated by scripts/write-data-pages.mjs -->
Common questions
How many codebases is this based on?
478 judged sessions across 2 realistic codebases. A category only runs on repositories where its seam is open, so coverage differs: some categories ran on more than ten codebases and some on two.
What eval tool do coding agents choose?
Across 478 judged sessions, Langfuse was chosen most often, in 31% of runs. Promptfoo was second with 15%. The result changes by who is asking.
Do Claude Code and Codex pick the same eval tool?
Yes. Claude Code and Codex both put Langfuse first in this category, which is less common than it sounds: they disagree on the leader in 16 of the 31 categories we measured.
How often do agents build evals themselves instead of installing something?
In 34% of runs the agent wrote the code itself rather than adopting a product. That makes hand-written code one of the strongest competitors in the category.
How can a vendor improve its position here?
Make the quickstart run when pasted, state the current version on the documentation page, use one name across product, package and import, write pages for the symptoms users describe rather than only the category name, and get into the repository through templates and framework integrations.
Which eval tools do agents consider but never choose?
Ragas (raised in 42 sessions, chosen in none), Traceloop (raised in 26 sessions, chosen in none). Being considered and not chosen is a different problem from being unknown, and it is usually fixable.
Where this comes from
Armature ran 15,287 judged sessions with Claude Code, Codex, Cursor, Grok Build CLI and Muse Code inside 110 realistic codebases, and published every run. The numbers on this page come from that work.