# How we measured this > The method behind every number here: 51 realistic codebases, 481 frozen prompts, real agents at pinned versions, and a blind judge. Source: https://armature.tech/library/how-we-measured-this Published: 2026-09-03 Publisher: Armature, Inc. (https://armature.tech) --- Every number in this library comes from one experiment. This page is the method, in one place, so the pages that use the numbers can point here instead of repeating it. ## The shape of it > **5,292 judged sessions** across **18 categories**, inside **51 realistic codebases**, driven by **481 frozen prompts** written as four kinds of person, run with **Claude Code, Codex and Cursor** at pinned versions in sandboxes, with a simulated project owner in the loop and a blind judge reading every session. Every run is published and replayable. The published set is 883 hours of agent time. The boards we ran and threw away are not in that number. ## Why you will see two numbers The essay is titled around **16,893** sessions and this page says **5,292**. Both are right and they count different things. 16,893 is every sandboxed session we ran. 5,292 is the subset that passed the checks and got published: valid runs, on the 51 codebases and 18 categories where the board holds up, each read by the judge. Roughly two runs in three did not make it. Some were quarantined when an agent version changed its output shape. Some belonged to boards that failed review and were re-run from scratch. Some were cells we cancelled once a wave had its first N valid runs, because which runs count is fixed at launch rather than chosen afterwards. Every number in this library is drawn from the 5,292. If you see a figure here that is a share or a count of sessions, that is the denominator. ## The codebases 75 repositories in the panel, 64 good enough to run, 51 in the published set. Ten languages: mostly TypeScript and JavaScript, then Python, Go, Java, C#, Ruby, PHP, and one each in Svelte and Rust. None are real companies. All of them have to look real to an agent that pokes around, so each belongs to a made-up company with a name, a README in that company's voice, and a git history that gets replayed when the cell is built. The history has fake authors with fake emails and commit messages like "scaffold next 14 storefront with tailwind". The lock files are real, made by real installs, and a linter checks every package hash against the live registry, because a hand-written lock file is the first thing an agent notices. The environment example files contain dead AWS keys, because a repository with no secrets in it does not look like a repository anyone uses. Everything is frozen by content hash. A wave pins a commit and a hash of the whole tree, and the agent receives a tarball built from exactly that. The metadata describing the repository is stripped before the agent sees it. One rule worth copying: a repository that mentions Armature, or another repository's fake company, fails the lint. The agent must never learn it is in a test. ## The open seams Each repository declares what it is missing. We call those [open seams](/library/open-seam), and a category's runs only go to repositories where its seam is open. That reduces coverage on purpose. Bot protection ran on five codebases, because the others had no public form worth protecting. Five valid repositories beats twenty invalid ones. ## The prompts 1,163 prompt files in about 170 families, 481 of them in the published set. Every request is written as one of four people: a vibe coder who describes symptoms and never names the category, a junior developer who names the category but never a vendor, a senior engineer who adds constraints, and a senior engineer at a large company who adds procurement rules. See [persona prompts](/library/persona-prompt). Each request also exists in several wordings, A, B and C, because wording moves the pick and we wanted to measure that rather than be fooled by it. The rules, each of which came from a mistake: - Every prompt adds a capability, never requests a migration, because a migration has an incumbent baked in. - The vibe coder may never say the category word. A linter fails the prompt if it does. - In the agent framework prompts, where the agents' own makers compete, the makers are always named in alternating order, and a linter checks it. In the published set, 78% of runs are the plain ask. The rest carry a theme such as self-hosting, volume and cost, or procurement. ## The runner Real command line tools, not an API that resembles them. Claude Code, Codex and Cursor, each pinned to one version, in a throwaway sandbox with two cores and four gigabytes of memory and a thirty-minute limit. The runner checks the tool's version before starting. It probes the network before and after, so we know the agent could reach the web. It records the starting commit, so a diff survives the agent committing its own work. And it refuses to count a run as finished unless the tool emitted its own final event: an exit code of zero with a truncated log is a crash, not a result. Every run is one row in Postgres. Runs execute once and are never retried. There is no delete permission on those tables. When one of these tools updates, its output format can change. A tool version whose event shape the parser does not recognise is quarantined, and every wave depending on it waits. That has cost us hours of throughput and it has never let a wrong result through. Which runs count is decided before any result exists. A wave takes the first N valid runs in an order fixed at launch, cancels the extras, and records a confidence interval. You cannot go back and keep your good runs. ## The simulated owner The first version of this had a problem: agents would end with "here are three good options, let me know which you prefer". For file storage, 10% of runs ended that way, and a leaderboard cannot count a shrug. So a small fast model plays the person who asked for help. The agent looks at the repository and recommends something. The owner answers with either one clarifying question or an approval. Only after approval does the agent write code. The owner gets at most two questions, and if it is still not satisfied the run fails rather than implement something unapproved. Its rules: it may not name a product the agent did not raise, it may not ask for a list of options, and it may not mention that any of this is a test. We check afterwards. In 15 runs out of 4,611, the owner slipped and named a product first. Those 15 are flagged on the public board. Across the published runs the owner asked 811 clarifying questions, pushed back 734 times, and in 255 runs demanded a specific product instead of a hedge. It approved every final answer. **Does the owner bias the result?** We checked, because it is the obvious objection. On bot protection, one product took 83% of runs without an owner and 70% with one. On file storage, Amazon S3 dropped from 65% to 57% while a challenger went from zero wins to six. The owner made leaders smaller, not bigger. It also multiplied the bill by about four. ## The judge A model reads every session and records what was chosen. It is a Gemini model, picked for one reason: it comes from a family that builds none of the three agents. A Claude judge can recognise Claude's prose and a GPT judge can recognise GPT's. See [judge model](/library/judge-model). The judge is blind. It sees a redacted transcript, the diff, the file tree and a frozen list of products. The owner's messages are relabelled as coming from "user". A tool name or model name leaking into the evidence is flagged as a leak. It produces more than a winner. For every product appearing in a session it records a role (primary pick, alternative, mention, rejected), a reason, a code from a fixed vocabulary, a pointer to the file or transcript line, and a confidence. About 76% of verdicts sit at full confidence. The judge is on version nine. Each version came from being wrong. Some of the rules: - One pick per product. Never "Prometheus slash Grafana". - A product named only inside a shell command still counts. - A proposed default the agent then builds on is a pick, at low confidence. - A vendor-neutral standard is never the winner. OpenTelemetry was mentioned in 382 runs and won zero, by rule, because it is the wiring and not the destination. With those rules, three copies of the judge agreed with each other 97.9% of the time. Without them, 84%. ## What we publish about the judge being wrong The judge undercounts. It misses names. So we keep a number called the judge gap: products the transcript mentions that the judge did not list. In observability, 155 of 360 runs have at least one. We publish that rather than pretending it is zero. ## The decoys Eight products in the registry do not exist. Two are planted in repositories. One lives on a fake wiki page our search engine can find. Two are inside an injected skill file. Two are in prompts. None are on any package registry and their domains are not registered. They are there to catch a specific failure: an agent, or a judge, confidently adopting something that cannot be adopted. ## What is not on the public pages The run traces and the transcripts. They are the raw material and they are served only inside the interactive board, behind a viewer gate. Every ranking, every breakdown and every key learning is public. ## The numbers as files The rankings are published as CSV and JSON under CC BY 4.0: [the dataset](/library/the-dataset). ## Replaying a run Every run in the published set can be opened and read on [the sector boards](/leaderboards/sectors). The interactive board carries the full trace for each one. ## Common questions ### How many sessions are behind these numbers? 5,292 judged sessions across 18 categories, run inside 51 realistic codebases with Claude Code, Codex and Cursor. Every run is published and can be replayed. ### Why do the repositories have to look real? Because an agent explores before it decides. A repository with a hand-written lock file, no git history and no README reads as artificial, and an agent that suspects a test behaves differently from one doing real work. ### What does the simulated project owner do? It plays the person who asked the agent for help. The agent recommends something, the owner either asks one clarifying question or approves, and only then does the agent write code. It stops a session ending in a shrug. ### Does the owner bias the result towards challengers? It made leaders smaller, not bigger. In bot protection one product took 83% of runs without an owner and 70% with one. In file storage, Amazon S3 went from 65% to 57% and a challenger went from zero wins to six. ### Why is the judge from a different model family than the agents? Because a model recognises its own family's prose. A Claude judge can identify a Claude session and a GPT judge a GPT session. The judge is a Gemini model, which builds none of the three agents tested. ## Read next - [How to measure install share](https://armature.tech/library/how-to-measure-install-share) (Markdown: https://armature.tech/library/how-to-measure-install-share.md) - [Judge model](https://armature.tech/library/judge-model) (Markdown: https://armature.tech/library/judge-model.md) - [Persona prompt](https://armature.tech/library/persona-prompt) (Markdown: https://armature.tech/library/persona-prompt.md) - [Open seam](https://armature.tech/library/open-seam) (Markdown: https://armature.tech/library/open-seam.md) - [The dataset: which software coding agents choose](https://armature.tech/library/the-dataset) (Markdown: https://armature.tech/library/the-dataset.md) --- Armature helps software products get discovered and used by coding agents. Service: https://armature.tech/discoverability · Results: https://armature.tech/leaderboards/sectors · Contact: contact@armature.tech