Do coding agents recommend OpenAI Evals?
Coding agents raised OpenAI Evals in 72 of 288 judged evals sessions and chose it in none of them. Measured with Claude Code, Codex and Cursor.
Coding agents raised OpenAI Evals in 72 of 288 judged evals sessions and chose it in none of them.
This page reports what happened when Claude Code, Codex and Cursor had to solve a problem in evals inside a realistic codebase. Not what a chat assistant says about OpenAI Evals. What an agent actually installed.
The numbers
| Category | Evals |
| Sessions in the category | 288 |
| Sessions where OpenAI Evals was chosen | 0 |
| Install share | 0% |
| Raised as a candidate, not chosen | 72 |
| Chosen when considered | 0% |
| Site | github.com |
Which agents raised OpenAI Evals
Every agent considered it and none adopted it. Cursor raised it most often, in 33% of its evals sessions.
| Agent | Sessions | Raised OpenAI Evals | Chose it |
|---|---|---|---|
| Claude Code | 96 | 25 (26%) | 0 |
| Codex | 96 | 15 (16%) | 0 |
| Cursor | 96 | 32 (33%) | 0 |
Which buyers it came up for
It surfaced most for requests written as senior engineer, in 30% of those sessions. Knowing which buyer already has the product in mind tells you which pages to fix first.
| Who is asking | Sessions | Raised it |
|---|---|---|
| Junior developer | 145 | 29 (20%) |
| Senior engineer | 143 | 43 (30%) |
What OpenAI Evals was up against
The full ranking in evals, from the same sessions:
| # | Product | Runs won | Share |
|---|---|---|---|
| 1 | Langfuse | 97 | 34% |
| 2 | Built in-house (no product adopted) | 84 | 29% |
| 3 | Promptfoo | 38 | 13% |
| 4 | Arize Phoenix | 23 | 8% |
| 5 | Braintrust | 18 | 6% |
| 6 | Inspect AI | 12 | 4% |
| 7 | LangSmith | 8 | 3% |
| 8 | Helicone | 6 | 2% |
What this means
Raised 72 times and chosen none is a specific, diagnosable result, and a better starting position than being unknown. The agent has the product in mind. It reached the shortlist 72 times. Something then lost every one of those sessions.
In our data the causes, in the order they occur:
- The quickstart does not run when pasted, so the agent abandoned it mid-integration.
- The documentation describes an interface that changed, so the generated code failed.
- The product name and the package name differ, so the install step went wrong.
- The fit was genuinely wrong for the repository, which is fine and worth knowing.
All but the last are fixable in days, and the reason is written down in the session transcript.
Where these numbers come from
The 288 sessions in evals are part of a published set of 5,292, run with real coding agents inside realistic codebases and judged blind. The full method is on one page: how we measured this.
Every evals run can be replayed on the board.
If you work on OpenAI Evals: the judge recorded a reason for every session where it was raised and passed over. Those reasons are in the transcripts.
<!-- generated by scripts/write-data-pages.mjs -->
Common questions
Do coding agents recommend OpenAI Evals?
They raise it but do not choose it. Across 72 sessions where OpenAI Evals came up as a candidate, agents chose something else every time.
Does Claude Code recommend OpenAI Evals?
In 0 of the 96 sessions in evals run with Claude Code, which is 0%.
Do different coding agents treat OpenAI Evals differently?
Not much. The three agents chose it at similar rates, between 0% and 0% of their runs.
How was this measured?
Real coding agents at pinned versions were run in sandboxes inside 51 realistic codebases and asked to solve real tasks. A simulated project owner approved or questioned each recommendation before any code was written, and a judge from a model family that builds none of the agents read every session blind.
Where this comes from
Armature ran 5,292 judged sessions with Claude Code, Codex and Cursor inside 51 realistic codebases, and published every run. The numbers on this page come from that work.