How to get picked for performance testing by coding agents
JMH took 8% of 359 judged performance testing sessions. What the numbers say a vendor in this category should do.
If you sell performance testing tools, this page is the part of the market no dashboard shows you: what a coding agent does when a developer asks for performance testing and never compares vendors.
The numbers come from 359 judged sessions with Claude Code, Codex, Cursor, Grok Build CLI and Muse Code, spread across 8 realistic codebases, with every session read by a judge.
What coding agents choose for performance testing
Across 359 judged sessions, JMH was chosen most often, in 8% of runs. pytest-benchmark was second with 6%. In 56% of runs the agent wrote the code itself and adopted no product at all.
| # | Product | Runs won | Share |
|---|---|---|---|
| 1 | Built in-house (no product adopted) | 202 | 56% |
| 2 | JMH | 30 | 8% |
| 3 | pytest-benchmark | 22 | 6% |
| 4 | Grafana k6 | 22 | 6% |
| 5 | hyperfine | 19 | 5% |
| 6 | PHPBench | 19 | 5% |
| 7 | Autocannon | 19 | 5% |
| 8 | BenchmarkDotNet | 8 | 2% |
| 9 | Benchstat | 4 | 1% |
| 10 | Tinybench | 4 | 1% |
Full board, every run replayable: the performance testing leaderboard.
What the shape of this category means
The leader takes only 8% of runs. This category is genuinely open and the ordering can be moved.
With the top product at 8%, performance testing is decided in the moment, from what the agent reads and what it finds in the repository. Nothing is locked in, which is the best situation a vendor can be in and the one where the work pays fastest.
The order here is set by the quality of what an agent can read and by whether your product is already present in the codebase. Both are things you can change.
The agents do not agree with each other
In this category Claude Code, Codex and Cursor did not put the same product first.
Grok Build CLI (18 sessions) and Muse Code (54 sessions) join this table with fewer runs, so read their rows as indicative.
| Agent | Runs | Picked most often |
|---|---|---|
| Claude Code | 89 | JMH (8) |
| Codex | 108 | Grafana k6 (18) |
| Cursor | 90 | JMH (7) |
| Grok Build CLI | 18 | Grafana k6 (2) |
| Muse Code | 54 | pytest-benchmark (3) |
That split decides where a vendor spends. In this category Codex ran a web search in 74% of its sessions and Claude Code in 0%, so what you publish reaches most of Codex's performance testing sessions and almost none of Claude Code's. Taking Grafana k6's position with Codex is mostly a content problem. Taking JMH's with Claude Code leans more on the repository.
Who is asking changes the answer
Every request in this experiment was written as a specific kind of person. In this category the leader changes with the person.
| Who is asking | Runs | Picked most often |
|---|---|---|
| Junior developer | 20 | null |
| Senior engineer | 239 | pytest-benchmark |
| Enterprise team | 100 | JMH |
That is 3 different products winning performance testing for 3 kinds of buyer, out of the same 359 sessions. Nobody here is winning performance testing. They are each winning one kind of buyer.
If you sell to more than one of them, you need pages for each. See how to win the enterprise persona.
What you are really competing against
In 56% of runs, the agent wrote the code itself. That makes hand-written code the largest single competitor in this category, larger than most vendors in the table above.
This changes the job of your content. Before you argue that you are better than another vendor, you have to argue that the problem is harder than it looks. What breaks at volume. Which edge cases cost a weekend. What the maintenance actually costs after six months.
That argument has to exist as a page an agent can read, with specifics and numbers. "It is harder than you think" is not an argument. "Here are the four failure modes and what each one costs" is.
Considered, and never chosen
Because the judge records every product an agent raised and not only the one it picked, this board also shows who kept reaching the shortlist and losing. In performance testing the clearest case is Criterion.rs: on the table in 50 sessions, chosen in none.
| Product | Raised in | Chosen in |
|---|---|---|
| Criterion.rs | 50 sessions | 0 |
| Lighthouse CI | 45 sessions | 0 |
| Locust | 34 sessions | 0 |
| github-action-benchmark | 27 sessions | 0 |
| iai-callgrind | 26 sessions | 0 |
Being rejected is a better position than being unknown, and a cheaper one to fix. The product is already in the agent's head and on the list. Whatever ended those 182 sessions is recorded in each transcript, one reason at a time.
What to do about it in performance testing
- Argue that the problem is harder than it looks, before you argue that you are better than a rival. 56% of performance testing sessions ended in hand-written code, so in roughly one session in 2 no vendor was in the running at all. Write the failure modes and the year-two maintenance cost, with numbers, in the documentation rather than the blog.
- Treat the ordering as movable. The leader holds 8%, so agents are deliberating rather than defaulting, and the inputs they use can change the answer. This is the most winnable shape a category comes in.
- Report install share per persona, not as one number. In performance testing the leader changes with who is asking, so a blended figure averages markets that behave differently.
- Measure per agent. Claude Code put JMH first, Codex put Grafana k6 first, Cursor put JMH first, Grok Build CLI put Grafana k6 first, Muse Code put pytest-benchmark first. A blended number for performance testing describes a market that does not exist.
The work that applies to every category rather than to this one is written up separately: audit your documentation, write a quickstart an agent can follow, and how to measure install share.
Every performance testing tool on this board
One page per product, with its install share, the per-agent split, and how often it was raised without being chosen.
- Do coding agents recommend JMH? — chosen in 8% of sessions
- Do coding agents recommend pytest-benchmark? — chosen in 6% of sessions
- Do coding agents recommend Grafana k6? — chosen in 6% of sessions
- Do coding agents recommend hyperfine? — chosen in 5% of sessions
- Do coding agents recommend PHPBench? — chosen in 5% of sessions
- Do coding agents recommend Autocannon? — chosen in 5% of sessions
- Do coding agents recommend BenchmarkDotNet? — chosen in 2% of sessions
Where these numbers come from
359 judged sessions in performance testing across 8 codebases, part of a published set of 15,287. Real coding agents at pinned versions, in sandboxes, inside realistic codebases, with a simulated project owner in the loop and a blind judge on every session. The full method is on one page: how we measured this.
Every performance testing run can be replayed on the board.
<!-- generated by scripts/write-data-pages.mjs -->
Common questions
How many codebases is this based on?
359 judged sessions across 8 realistic codebases. A category only runs on repositories where its seam is open, so coverage differs: some categories ran on more than ten codebases and some on two.
What performance testing tool do coding agents choose?
Across 359 judged sessions, JMH was chosen most often, in 8% of runs. pytest-benchmark was second with 6%. The result changes by agent and by who is asking.
Do Claude Code and Codex pick the same performance testing tool?
No. Claude Code picked JMH, Codex picked Grafana k6, Cursor picked JMH. Measuring one agent tells you about part of the market only.
How often do agents build performance testing themselves instead of installing something?
In 56% of runs the agent wrote the code itself rather than adopting a product. That makes hand-written code one of the strongest competitors in the category.
How can a vendor improve its position here?
Make the quickstart run when pasted, state the current version on the documentation page, use one name across product, package and import, write pages for the symptoms users describe rather than only the category name, and get into the repository through templates and framework integrations.
Which performance testing tools do agents consider but never choose?
Criterion.rs (raised in 50 sessions, chosen in none), Lighthouse CI (raised in 45 sessions, chosen in none), Locust (raised in 34 sessions, chosen in none), github-action-benchmark (raised in 27 sessions, chosen in none), iai-callgrind (raised in 26 sessions, chosen in none). Being considered and not chosen is a different problem from being unknown, and it is usually fixable.
Where this comes from
Armature ran 15,287 judged sessions with Claude Code, Codex, Cursor, Grok Build CLI and Muse Code inside 110 realistic codebases, and published every run. The numbers on this page come from that work.