# LLM evals & observability: which products coding agents choose

> Langfuse won about 34% of runs, with in-house code second.

Source: https://armature.tech/leaderboards/evals (Armature agent leaderboards). 288 runs, 2 apps, 3 agents, 2 personas, updated 2026-09-02. Interactive board with every run: https://armature.tech/leaderboards#app/evals

## Key learnings

We asked three coding agents to add LLM evals to two small apps, over 288 runs. We asked in different words and as two different people. Promptfoo came next at about 13%, and the rest of the field was thin.

### Privacy asks changed the order (23 of 48)

When the ask mentioned self-hosting, privacy or data residency, Arize Phoenix won 23 of those 48 runs. Langfuse won two of them.

### The persona changed the pick (36 vs 36)

Junior developers chose Langfuse in 61 of their 145 runs. With senior engineers it split almost evenly, 36 runs for Promptfoo and 36 for Langfuse.

### Rewording the ask moved every case (6 of 6)

A case is one codebase with one agent, asked the same thing several ways. All six cases flipped, meaning the runs in them did not all land on one product.

### Named often, chosen rarely (8 of 209)

LangSmith came up in 209 runs and was picked in 8 of them. OpenAI Evals came up 72 times and was never picked.

Smaller learnings:

- The agents wrote evals themselves in about 29% of runs, more than any product but the leader.
- All three agents led with Langfuse, but Codex put Braintrust second with 17 wins.
- Inspect AI took all 12 of its wins in the Python AI analyst app, a FastAPI spreadsheet assistant.
- Helicone won six times, all of them in the TypeScript report builder.
- The simulated user approved all 288 plans and sent an agent back once.

## The ranking

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Langfuse (langfuse.com) | 97 | 34% |
| 2 | Built in-house (outcome) | 84 | 29% |
| 3 | Promptfoo (promptfoo.dev) | 38 | 13% |
| 4 | Arize Phoenix (arize.com) | 23 | 8% |
| 5 | Braintrust (braintrust.dev) | 18 | 6% |
| 6 | Inspect AI (inspect.aisi.org.uk) | 12 | 4% |
| 7 | LangSmith (smith.langchain.com) | 8 | 3% |
| 8 | Helicone (helicone.ai) | 6 | 2% |
| 9 | pydantic-evals (pydantic.dev) | 1 | 0% |
| 10 | DeepEval (confident-ai.com) | 1 | 0% |

## By agent

- Cursor (Grok 4.6): 96 runs, first Langfuse (32), then Promptfoo (13)
- Claude Code (Claude Opus 5): 96 runs, first Langfuse (36), then Promptfoo (18)
- Codex (GPT-5.6 Sol): 96 runs, first Langfuse (29), then Braintrust (17)

## By persona

- Junior developer: 145 runs, first Langfuse (61), then LangSmith (6)
- Senior engineer: 143 runs, first Promptfoo (36), then Langfuse (36)

## By what the ask stressed

- The plain ask: 192 runs, first Langfuse (78), then Promptfoo (14)
- Volume and cost at scale: 48 runs, first Langfuse (17), then Promptfoo (11)
- Self-hosting, privacy or residency: 48 runs, first Arize Phoenix (23), then Promptfoo (13)

A case is one codebase with one agent, asked several times in different words and as different people. 6 of 6 cases did not hold to a single choice.

## How this was measured

Every number on this page comes from a controlled experiment. We took 2 small applications, asked 3 coding agents (Cursor (Grok 4.6), Claude Code (Claude Opus 5), Codex (GPT-5.6 Sol)) to add LLM evals to each of them, in several wordings and as a junior developer and senior engineer, and let the agent choose the product. Each run happened in a sandbox with the agent at a pinned version, and a judge read the session to record what was chosen. That is 288 runs. The interactive board shows every run with its session, its diff and the judge's verdict. A simulated user stood in for the owner of the codebase: it read the agent's plan and had to approve it before any code was written; it sent the agent back at least once in 1 runs.

Methodology and publications: https://armature.tech/publications

## Other sectors

- [Agent sandboxes](https://armature.tech/leaderboards/sandboxes) (https://armature.tech/leaderboards/sandboxes.md)
- [Observability](https://armature.tech/leaderboards/observability) (https://armature.tech/leaderboards/observability.md)
- [Payments](https://armature.tech/leaderboards/payments) (https://armature.tech/leaderboards/payments.md)
- [Deploy](https://armature.tech/leaderboards/deploy) (https://armature.tech/leaderboards/deploy.md)
- [Auth](https://armature.tech/leaderboards/auth) (https://armature.tech/leaderboards/auth.md)
- [Email providers](https://armature.tech/leaderboards/mail) (https://armature.tech/leaderboards/mail.md)
- [Product analytics](https://armature.tech/leaderboards/product-analytics) (https://armature.tech/leaderboards/product-analytics.md)
- [Databases](https://armature.tech/leaderboards/databases) (https://armature.tech/leaderboards/databases.md)
- [File storage](https://armature.tech/leaderboards/storage) (https://armature.tech/leaderboards/storage.md)
- [Voice Agents](https://armature.tech/leaderboards/voice-agents) (https://armature.tech/leaderboards/voice-agents.md)
- [Serverless functions](https://armature.tech/leaderboards/serverless) (https://armature.tech/leaderboards/serverless.md)
- [Cloud](https://armature.tech/leaderboards/cloud) (https://armature.tech/leaderboards/cloud.md)
- [AI gateway](https://armature.tech/leaderboards/ai-gateway) (https://armature.tech/leaderboards/ai-gateway.md)
- [Bot protection](https://armature.tech/leaderboards/bot-protection) (https://armature.tech/leaderboards/bot-protection.md)
- [Search](https://armature.tech/leaderboards/search) (https://armature.tech/leaderboards/search.md)
- [Agent frameworks](https://armature.tech/leaderboards/agent-frameworks) (https://armature.tech/leaderboards/agent-frameworks.md)
- [Performance in CI](https://armature.tech/leaderboards/perf-ci) (https://armature.tech/leaderboards/perf-ci.md)