# Do coding agents recommend pytest-benchmark? > pytest-benchmark was chosen in 7% of 216 judged performance testing sessions, ranking second. Measured with Claude Code, Codex and Cursor. Source: https://armature.tech/library/do-coding-agents-recommend-pytest-benchmark Published: 2026-09-03 Publisher: Armature, Inc. (https://armature.tech) --- > pytest-benchmark was chosen in 7% of 216 judged performance testing sessions, ranking second. It was also raised as a candidate in 25 further sessions without being chosen. This page reports what happened when Claude Code, Codex and Cursor had to solve a problem in performance testing inside a realistic codebase. Not what a chat assistant says about pytest-benchmark. What an agent actually installed. One thing to read first: every one of those wins came from a **single codebase**. That is a result about one repository rather than about performance testing in general, and the splits below cannot separate the two. Treat them as a description of that repository. ## The numbers | | | | --- | --- | | Category | Performance testing | | Sessions in the category | 216 | | Sessions where pytest-benchmark was chosen | 15 | | Install share | 7% | | Rank in category | 2 of 14 | | Codebases it won in | 1 | | Raised as a candidate, not chosen | 25 | | Chosen when considered | 38% | | Site | pytest-benchmark.readthedocs.io | ## By agent With 15 wins spread across three agents, the rates below are small numbers and a difference between them is not yet a finding. They are here because the direction is worth knowing, not because the gap is established. | Agent | Sessions | Chose pytest-benchmark | Share | | --- | --- | --- | --- | | Claude Code | 72 | 5 | 7% | | Codex | 72 | 6 | 8% | | Cursor | 72 | 4 | 6% | ## By who was asking pytest-benchmark performs similarly across the four kinds of buyer, from 0% to 10%. That is unusual: the category leader changed with the persona in 14 of the 18 categories we measured. | Who is asking | Sessions | Chose pytest-benchmark | Share | | --- | --- | --- | --- | | Junior developer | 12 | 0 | 0% | | Senior engineer | 144 | 15 | 10% | | Enterprise team | 60 | 0 | 0% | ## What pytest-benchmark was up against The full ranking in performance testing, from the same sessions: | # | Product | Runs won | Share | | --- | --- | --- | --- | | 1 | Built in-house (no product adopted) | 113 | 52% | | 2 | JMH | 22 | 10% | | 3 | pytest-benchmark **(this page)** | 15 | 7% | | 4 | Autocannon | 14 | 6% | | 5 | hyperfine | 13 | 6% | | 6 | PHPBench | 12 | 6% | | 7 | Grafana k6 | 8 | 4% | | 8 | BenchmarkDotNet | 7 | 3% | ## What this means pytest-benchmark was raised in 25 sessions and chosen in 15. That ratio is balanced enough that the ceiling is presence rather than integration: the product converts reasonably when it is on the table, and it is not on the table often enough. ## Where these numbers come from The 216 sessions in performance testing are part of a published set of 5,292, run with real coding agents inside realistic codebases and judged blind. The full method is on one page: [how we measured this](/library/how-we-measured-this). Every performance testing run can be replayed on [the board](/leaderboards/perf-ci). If you work on pytest-benchmark: the judge recorded a reason for every session where it was raised and passed over. Those reasons are in the transcripts. ## Common questions ### Do coding agents recommend pytest-benchmark? Yes. pytest-benchmark was chosen in 15 of the 216 judged sessions in performance testing, a 7% install share, ranking second in its category. ### Does Claude Code recommend pytest-benchmark? In 5 of the 72 sessions in performance testing run with Claude Code, which is 7%. ### Do different coding agents treat pytest-benchmark differently? Not much. The three agents chose it at similar rates, between 6% and 8% of their runs. ### How was this measured? Real coding agents at pinned versions were run in sandboxes inside 51 realistic codebases and asked to solve real tasks. A simulated project owner approved or questioned each recommendation before any code was written, and a judge from a model family that builds none of the agents read every session blind. ### How often is pytest-benchmark considered but not chosen? It was raised as a candidate in 25 sessions without being chosen, and chosen in 15. That is a 38% conversion from considered to chosen. ## Read next - [How to get picked for performance testing by coding agents](https://armature.tech/library/perf-ci-coding-agents-playbook) (Markdown: https://armature.tech/library/perf-ci-coding-agents-playbook.md) - [Do coding agents recommend JMH?](https://armature.tech/library/do-coding-agents-recommend-jmh) (Markdown: https://armature.tech/library/do-coding-agents-recommend-jmh.md) - [Do coding agents recommend Autocannon?](https://armature.tech/library/do-coding-agents-recommend-autocannon) (Markdown: https://armature.tech/library/do-coding-agents-recommend-autocannon.md) - [Do Claude Code, Codex and Cursor pick the same tools?](https://armature.tech/library/do-claude-code-and-codex-agree) (Markdown: https://armature.tech/library/do-claude-code-and-codex-agree.md) --- Armature helps software products get discovered and used by coding agents. Service: https://armature.tech/discoverability · Results: https://armature.tech/leaderboards/sectors · Contact: contact@armature.tech