# Performance in CI: which products coding agents choose

> Agents wrote the gate themselves in about 52% of runs.

Source: https://armature.tech/leaderboards/perf-ci (Armature agent leaderboards). 216 runs, 8 apps, 3 agents, 3 personas, updated 2026-09-01. Interactive board with every run: https://armature.tech/leaderboards#app/perf-ci

## Key learnings

We asked three coding agents to add a performance gate to eight small apps, 216 runs in all. We varied the wording and asked as three different people. Among the named products JMH came first, at about 10% of runs.

### The codebase language picked the tool (13 wins)

hyperfine won 13 times, all in the Rust CLI tool. PHPBench took 12, all in the PHP government portal. The seven wins for BenchmarkDotNet were all in the .NET field ops app.

### Most cases did not settle on one product (18 of 24)

A case here is one codebase with one agent, asked several times in different words. In 18 of 24 cases the runs did not all land on the same product.

### One persona carried the leader (22 of 60)

The enterprise team ran 60 times, and all 22 wins for JMH came there. Under senior engineers, pytest-benchmark led with 15.

### Named in many plans, chosen in few (3 of 121)

CodSpeed came up 121 times and was picked three times. Bencher came up 83 times and was picked once.

Smaller learnings:

- The simulated user approved all 216 plans, and sent the agent back at least once in 13 of them.
- Codex picked JMH 11 times, more than any other agent picked any product.
- Claude Code split its top pick evenly, five wins each for Autocannon and JMH.
- The junior developer ran only 12 times, and no product came out on top there.

## The ranking

| # | Product | Wins | Share |
|---|---|---:|---:|
| 1 | Built in-house (outcome) | 113 | 52% |
| 2 | JMH (openjdk.org) | 22 | 10% |
| 3 | pytest-benchmark (pytest-benchmark.readthedocs.io) | 15 | 7% |
| 4 | Autocannon (github.com) | 14 | 6% |
| 5 | hyperfine (github.com) | 13 | 6% |
| 6 | PHPBench (phpbench.readthedocs.io) | 12 | 6% |
| 7 | Grafana k6 (k6.io) | 8 | 4% |
| 8 | BenchmarkDotNet (benchmarkdotnet.org) | 7 | 3% |
| 9 | CodSpeed (codspeed.io) | 3 | 1% |
| 10 | Tinybench (github.com) | 3 | 1% |
| 11 | Vitest Bench (vitest.dev) | 2 | 1% |
| 12 | perfgate (github.com) | 1 | 0% |
| 13 | Benchstat (github.com) | 1 | 0% |
| 14 | Gungraun (github.com) | 1 | 0% |
| 15 | Bencher (bencher.dev) | 1 | 0% |

## By agent

- Cursor (Grok 4.6): 72 runs, first JMH (6), then pytest-benchmark (4)
- Claude Code (Claude Opus 5): 72 runs, first Autocannon (5), then JMH (5)
- Codex (GPT-5.6 Sol): 72 runs, first JMH (11), then hyperfine (7)

## By persona

- Senior engineer: 144 runs, first pytest-benchmark (15), then Autocannon (14)
- Enterprise team: 60 runs, first JMH (22), then PHPBench (12)
- Junior developer: 12 runs, first none

## By what the ask stressed

- The plain ask: 216 runs, first JMH (22), then pytest-benchmark (15)

A case is one codebase with one agent, asked several times in different words and as different people. 18 of 24 cases did not hold to a single choice.

## How this was measured

Every number on this page comes from a controlled experiment. We took 8 small applications, asked 3 coding agents (Cursor (Grok 4.6), Claude Code (Claude Opus 5), Codex (GPT-5.6 Sol)) to add a performance gate to each of them, in several wordings and as a senior engineer and enterprise team and junior developer, and let the agent choose the product. Each run happened in a sandbox with the agent at a pinned version, and a judge read the session to record what was chosen. That is 216 runs. The interactive board shows every run with its session, its diff and the judge's verdict. A simulated user stood in for the owner of the codebase: it read the agent's plan and had to approve it before any code was written; it sent the agent back at least once in 13 runs.

Methodology and publications: https://armature.tech/publications

## Other sectors

- [Agent sandboxes](https://armature.tech/leaderboards/sandboxes) (https://armature.tech/leaderboards/sandboxes.md)
- [Observability](https://armature.tech/leaderboards/observability) (https://armature.tech/leaderboards/observability.md)
- [Payments](https://armature.tech/leaderboards/payments) (https://armature.tech/leaderboards/payments.md)
- [Deploy](https://armature.tech/leaderboards/deploy) (https://armature.tech/leaderboards/deploy.md)
- [Auth](https://armature.tech/leaderboards/auth) (https://armature.tech/leaderboards/auth.md)
- [Email providers](https://armature.tech/leaderboards/mail) (https://armature.tech/leaderboards/mail.md)
- [Product analytics](https://armature.tech/leaderboards/product-analytics) (https://armature.tech/leaderboards/product-analytics.md)
- [Databases](https://armature.tech/leaderboards/databases) (https://armature.tech/leaderboards/databases.md)
- [File storage](https://armature.tech/leaderboards/storage) (https://armature.tech/leaderboards/storage.md)
- [LLM evals & observability](https://armature.tech/leaderboards/evals) (https://armature.tech/leaderboards/evals.md)
- [Voice Agents](https://armature.tech/leaderboards/voice-agents) (https://armature.tech/leaderboards/voice-agents.md)
- [Serverless functions](https://armature.tech/leaderboards/serverless) (https://armature.tech/leaderboards/serverless.md)
- [Cloud](https://armature.tech/leaderboards/cloud) (https://armature.tech/leaderboards/cloud.md)
- [AI gateway](https://armature.tech/leaderboards/ai-gateway) (https://armature.tech/leaderboards/ai-gateway.md)
- [Bot protection](https://armature.tech/leaderboards/bot-protection) (https://armature.tech/leaderboards/bot-protection.md)
- [Search](https://armature.tech/leaderboards/search) (https://armature.tech/leaderboards/search.md)
- [Agent frameworks](https://armature.tech/leaderboards/agent-frameworks) (https://armature.tech/leaderboards/agent-frameworks.md)