Agent leaderboards / All sectors / Performance in CI

Performance in CI: which products coding agents choose

Agents built the gate in-house in about 56% of runs.

359 runs8 apps5 agents3 personasupdated 2026-09-29

The interactive board, open on performance in ci. Open it full page · this page as Markdown

Read this leaderboard as textrankings, key learnings, method

Key learnings

We asked five agents to add a performance gate to 8 codebases, 359 runs in all. We asked in different words and as three kinds of person. Most of the time the agents wrote the gate themselves.

30 of 359

No product got past 8%

JMH led the products with 30 wins. pytest-benchmark and Grafana k6 came next, almost evenly, with 22 each. The counts fall away quickly after that.

19 wins

Four products appeared in one codebase each

hyperfine took 19 wins, all in the Rust CLI tool. PHPBench also took 19, all in the PHP government portal. BenchmarkDotNet showed up only in the .NET field ops app, and Benchstat only in the Go fleet tracker.

31 of 39

Rewording the ask changed what came back

A case here is one codebase with one agent, asked several times in different words and as different people. In 31 of 39 cases those runs did not all land on the same product.

3 of 157

Two products were named often and picked rarely

CodSpeed came up in 157 runs and was chosen in three. Bencher came up in 118 runs and was chosen once.

  • Codex picked Grafana k6 18 times, more than it picked anything else.
  • As enterprise teams the agents took JMH 30 times; as senior engineers they most often took pytest-benchmark.
  • The simulated user approved every plan, though in 19 runs it sent the agent back first.
Explore every run in the interactive board

The ranking359 runs

ProductWinsShare
1 Built in-houseoutcome 202 56%
2 JMHopenjdk.org 30 8%
3 pytest-benchmarkpytest-benchmark.readthedocs.io 22 6%
4 Grafana k6k6.io 22 6%
5 hyperfinegithub.com 19 5%
6 PHPBenchphpbench.readthedocs.io 19 5%
7 Autocannongithub.com 19 5%
8 BenchmarkDotNetbenchmarkdotnet.org 8 2%
9 Benchstatgithub.com 4 1%
10 Tinybenchgithub.com 4 1%
11 CodSpeedcodspeed.io 3 1%
12 Vitest Benchvitest.dev 2 1%
13 perfgategithub.com 1 0%
14 Gungraungithub.com 1 0%
15 Bencherbencher.dev 1 0%
16 NBombernbomber.com 1 0%
17 Gatlinggatling.io 1 0%

By agent, by persona, by wording

By agent

Codex108 runsGrafana k6 · 18then JMH · 12
Cursor90 runsJMH · 7then pytest-benchmark · 6
Claude Code89 runsJMH · 8then Autocannon · 6
Muse Code · Muse Spark 1.354 runspytest-benchmark · 3then JMH · 2
Grok Build CLI · Grok 4.718 runsGrafana k6 · 2then JMH · 1

By persona

Senior engineer239 runspytest-benchmark · 22then hyperfine · 19
Enterprise team100 runsJMH · 30then PHPBench · 19
Junior developer20 runs

By what the ask stressed

The plain ask359 runsJMH · 30then pytest-benchmark · 22

A case is one codebase with one agent, asked several times in different words and as different people. 31 of 39 cases did not hold to a single product.

How this was measured

Every number on this page comes from a controlled experiment. We took 8 small applications, asked 5 coding agents (Codex, Cursor, Claude Code, Muse Code (Muse Spark 1.3), Grok Build CLI (Grok 4.7)) to add a performance gate to each of them, in several wordings and as a senior engineer and enterprise team and junior developer, and let the agent choose the product. Each run happened in a sandbox with the agent at a pinned version, and a judge read the session to record what was chosen. That is 359 runs. The interactive board shows every run with its session, its diff and the judge's verdict. A simulated user stood in for the owner of the codebase: it read the agent's plan and had to approve it before any code was written; it sent the agent back at least once in 19 runs. Read the methodology and the publications.

If you sell in this sector: what these numbers mean for a vendor.

Open the interactive boardThis page as Markdown

Other sectors