How to get picked for performance testing by coding agents
JMH took 10% of 216 judged performance testing sessions. What the numbers say a vendor in this category should do.
If you sell performance testing tools, this page is the part of the market no dashboard shows you: what a coding agent does when a developer asks for performance testing and never compares vendors.
The numbers come from 216 judged sessions with Claude Code, Codex and Cursor, spread across 8 realistic codebases, with every session read by a judge.
What coding agents choose for performance testing
Across 216 judged sessions, JMH was chosen most often, in 10% of runs. pytest-benchmark was second with 7%. In 52% of runs the agent wrote the code itself and adopted no product at all.
| # | Product | Runs won | Share |
|---|---|---|---|
| 1 | Built in-house (no product adopted) | 113 | 52% |
| 2 | JMH | 22 | 10% |
| 3 | pytest-benchmark | 15 | 7% |
| 4 | Autocannon | 14 | 6% |
| 5 | hyperfine | 13 | 6% |
| 6 | PHPBench | 12 | 6% |
| 7 | Grafana k6 | 8 | 4% |
| 8 | BenchmarkDotNet | 7 | 3% |
| 9 | CodSpeed | 3 | 1% |
| 10 | Tinybench | 3 | 1% |
Full board, every run replayable: the performance testing leaderboard.
What the shape of this category means
The leader takes only 10% of runs. This category is genuinely open and the ordering can be moved.
With the top product at 10%, performance testing is decided in the moment, from what the agent reads and what it finds in the repository. Nothing is locked in, which is the best situation a vendor can be in and the one where the work pays fastest.
The order here is set by the quality of what an agent can read and by whether your product is already present in the codebase. Both are things you can change.
The agents do not agree with each other
In this category the three agents we ran put different products first.
| Agent | Runs | Picked most often |
|---|---|---|
| Claude Code | 72 | Autocannon (5) |
| Codex | 72 | JMH (11) |
| Cursor | 72 | JMH (6) |
That split decides where a vendor spends. Codex ran a web search in 53% of decision runs and Claude Code in 1.6%, so the pages you publish are live in half of Codex's performance testing sessions and almost none of Claude Code's. Taking JMH's position with Codex is a content problem. Taking Autocannon's with Claude Code is a repository problem.
Who is asking changes the answer
Every request in this experiment was written as a specific kind of person. In this category the leader changes with the person.
| Who is asking | Runs | Picked most often |
|---|---|---|
| Junior developer | 12 | null |
| Senior engineer | 144 | pytest-benchmark |
| Enterprise team | 60 | JMH |
That is 3 different products winning performance testing for 3 kinds of buyer, out of the same 216 sessions. Nobody here is winning performance testing. They are each winning one kind of buyer.
If you sell to more than one of them, you need pages for each. See how to win the enterprise persona.
What you are really competing against
In 52% of runs, the agent wrote the code itself. That makes hand-written code the largest single competitor in this category, larger than most vendors in the table above.
This changes the job of your content. Before you argue that you are better than another vendor, you have to argue that the problem is harder than it looks. What breaks at volume. Which edge cases cost a weekend. What the maintenance actually costs after six months.
That argument has to exist as a page an agent can read, with specifics and numbers. "It is harder than you think" is not an argument. "Here are the four failure modes and what each one costs" is.
Considered, and never chosen
Because the judge records every product an agent raised and not only the one it picked, this board also shows who kept reaching the shortlist and losing. In performance testing the clearest case is Criterion.rs: on the table in 31 sessions, chosen in none.
| Product | Raised in | Chosen in |
|---|---|---|
| Criterion.rs | 31 sessions | 0 |
| Lighthouse CI | 30 sessions | 0 |
| Gatling | 24 sessions | 0 |
| github-action-benchmark | 23 sessions | 0 |
| iai-callgrind | 21 sessions | 0 |
Being rejected is a better position than being unknown, and a cheaper one to fix. The product is already in the agent's head and on the list. Whatever ended those 129 sessions is recorded in each transcript, one reason at a time.
What to do about it in performance testing
- Argue that the problem is harder than it looks, before you argue that you are better than a rival. 52% of performance testing sessions ended in hand-written code, so in roughly one session in 2 no vendor was in the running at all. Write the failure modes and the year-two maintenance cost, with numbers, in the documentation rather than the blog.
- Treat the ordering as movable. The leader holds 10%, so agents are deliberating rather than defaulting, and the inputs they use can change the answer. This is the most winnable shape a category comes in.
- Report install share per persona, not as one number. In performance testing the leader changes with who is asking, so a blended figure averages markets that behave differently.
- Measure per agent. Claude Code put Autocannon first, Codex put JMH first, Cursor put JMH first. A blended number for performance testing describes a market that does not exist.
The work that applies to every category rather than to this one is written up separately: audit your documentation, write a quickstart an agent can follow, and how to measure install share.
Every performance testing tool on this board
One page per product, with its install share, the per-agent split, and how often it was raised without being chosen.
- Do coding agents recommend JMH? — chosen in 10% of sessions
- Do coding agents recommend pytest-benchmark? — chosen in 7% of sessions
- Do coding agents recommend Autocannon? — chosen in 6% of sessions
- Do coding agents recommend hyperfine? — chosen in 6% of sessions
- Do coding agents recommend PHPBench? — chosen in 6% of sessions
- Do coding agents recommend Grafana k6? — chosen in 4% of sessions
Where these numbers come from
216 judged sessions in performance testing across 8 codebases, part of a published set of 5,292. Real coding agents at pinned versions, in sandboxes, inside realistic codebases, with a simulated project owner in the loop and a blind judge on every session. The full method is on one page: how we measured this.
Every performance testing run can be replayed on the board.
<!-- generated by scripts/write-data-pages.mjs -->
Common questions
How many codebases is this based on?
216 judged sessions across 8 realistic codebases. A category only runs on repositories where its seam is open, so coverage differs: some categories ran on more than ten codebases and some on two.
What performance testing tool do coding agents choose?
Across 216 judged sessions, JMH was chosen most often, in 10% of runs. pytest-benchmark was second with 7%. The result changes by agent and by who is asking.
Do Claude Code and Codex pick the same performance testing tool?
No. Claude Code picked Autocannon, Codex picked JMH, Cursor picked JMH. Measuring one agent tells you about part of the market only.
How often do agents build performance testing themselves instead of installing something?
In 52% of runs the agent wrote the code itself rather than adopting a product. That makes hand-written code one of the strongest competitors in the category.
How can a vendor improve its position here?
Make the quickstart run when pasted, state the current version on the documentation page, use one name across product, package and import, write pages for the symptoms users describe rather than only the category name, and get into the repository through templates and framework integrations.
Which performance testing tools do agents consider but never choose?
Criterion.rs (raised in 31 sessions, chosen in none), Lighthouse CI (raised in 30 sessions, chosen in none), Gatling (raised in 24 sessions, chosen in none), github-action-benchmark (raised in 23 sessions, chosen in none), iai-callgrind (raised in 21 sessions, chosen in none). Being considered and not chosen is a different problem from being unknown, and it is usually fixable.
Where this comes from
Armature ran 5,292 judged sessions with Claude Code, Codex and Cursor inside 51 realistic codebases, and published every run. The numbers on this page come from that work.