Agent leaderboards / All sectors / LLM evals & observability

LLM evals & observability: which products coding agents choose

Agents wrote their own evals in about 34% of runs.

478 runs2 apps5 agents2 personasupdated 2026-09-29

The interactive board, open on llm evals & observability. Open it full page · this page as Markdown

Read this leaderboard as textrankings, key learnings, method

Key learnings

We asked five coding agents to add LLM evals to two codebases, 478 times in all. We varied the wording, and asked as two different people. Writing it themselves came out on top, and Langfuse led the named products at about 31%.

92 vs 11

The persona changed which product came first

Junior developers picked Langfuse 92 times, against 11 for Promptfoo. With senior engineers the two split almost evenly, 59 to 56 in favor of Promptfoo.

32 of 79

Self-hosting asks brought a different winner

When the ask mentioned self-hosting, privacy or residency, Arize Phoenix won 32 of 79 runs. Every win it took in this sector came from that theme.

10 of 10

The same question in new words got new answers

A case here is one codebase with one agent, asked several times in different words. All 10 cases flipped.

318 vs 9

Named often, chosen rarely

LangSmith came up 318 times while the agents compared options. It was picked 9 times.

  • The simulated user approved all 478 plans, and sent the agent back at least once in three runs.
  • Inspect AI took all 16 of its wins in the Python AI analyst codebase.
  • Helicone won 12 times, every one of them in the AI report builder.
  • Codex is the only agent whose second pick was Braintrust, at 22 wins.
Explore every run in the interactive board

The ranking478 runs

ProductWinsShare
1 Built in-houseoutcome 161 34%
2 Langfuselangfuse.com 148 31%
3 Promptfoopromptfoo.dev 70 15%
4 Arize Phoenixarize.com 32 7%
5 Braintrustbraintrust.dev 23 5%
6 Inspect AIinspect.aisi.org.uk 16 3%
7 Heliconehelicone.ai 12 3%
8 LangSmithsmith.langchain.com 9 2%
9 pydantic-evalspydantic.dev 4 1%
10 DeepEvalconfident-ai.com 1 0%
11 OpenAI Evalsgithub.com 1 0%
12 Axiomaxiom.co 1 0%

By agent, by persona, by wording

By agent

Codex144 runsLangfuse · 48then Braintrust · 22
Cursor120 runsLangfuse · 40then Promptfoo · 19
Claude Code120 runsLangfuse · 46then Promptfoo · 22
Muse Code · Muse Spark 1.372 runsLangfuse · 8then Promptfoo · 7
Grok Build CLI · Grok 4.722 runsLangfuse · 6then Promptfoo · 5

By persona

Junior developer241 runsLangfuse · 92then Promptfoo · 11
Senior engineer237 runsPromptfoo · 59then Langfuse · 56

By what the ask stressed

The plain ask319 runsLangfuse · 119then Promptfoo · 28
Volume and cost at scale80 runsLangfuse · 26then Promptfoo · 18
Self-hosting, privacy or residency79 runsArize Phoenix · 32then Promptfoo · 24

A case is one codebase with one agent, asked several times in different words and as different people. 10 of 10 cases did not hold to a single product.

How this was measured

Every number on this page comes from a controlled experiment. We took 2 small applications, asked 5 coding agents (Codex, Cursor, Claude Code, Muse Code (Muse Spark 1.3), Grok Build CLI (Grok 4.7)) to add LLM evals to each of them, in several wordings and as a junior developer and senior engineer, and let the agent choose the product. Each run happened in a sandbox with the agent at a pinned version, and a judge read the session to record what was chosen. That is 478 runs. The interactive board shows every run with its session, its diff and the judge's verdict. A simulated user stood in for the owner of the codebase: it read the agent's plan and had to approve it before any code was written; it sent the agent back at least once in 3 runs. Read the methodology and the publications.

If you sell in this sector: what these numbers mean for a vendor.

Open the interactive boardThis page as Markdown

Other sectors