Which tools do coding agents install? What 3,547 judged runs across 18 sectors show

Coding agents now make a decision that used to belong to a developer: which tool gets installed. Ask Claude Code to add error tracking to a repository and it picks a product, installs it and writes the integration. Nobody clicks a pricing page.

We wanted to know what these agents choose, and why. Public benchmarks so far ask an agent an open question in an empty folder and record the answer. Real agents work inside a repository, on a task, with a language, a framework and a cloud already in place. So we built a way to measure the choice in that setting, at scale, and we published every run.

This article explains what we built, what we found, and where the method is weak. The results live on the leaderboards, and every number there opens the full session behind it.

What we built

A panel of 51 repositories. Each one is a fictional company with a real codebase: a ticketing API on Express, a clinic roster on Flask, a field operations app on .NET, a telecom back office on Java with Splunk, and so on. They cover eight languages and many stacks. They have synthetic git history, well-formed but fake credentials, and documentation that reads like a small team wrote it. An agent that explores one of them sees a real project, not a benchmark. Every run starts from the same frozen commit.

481 frozen prompts, written as personas. Every prompt asks to add a capability to the repository. We wrote them as four kinds of people: a vibe coder who describes symptoms and never names the category, a junior developer who names the category but no vendor, a senior engineer who adds constraints, and an enterprise team that adds policy. Prompts are frozen, so a run from August and a run from September are comparable.

Two coding agents, at pinned versions. We run the real Claude Code and Codex command line tools, not an API call that imitates them. Claude Code ran Opus 5 and Sonnet 5. Codex ran GPT-5.6 Sol and GPT-5.6 Terra. The agent does the whole job. It searches, picks, installs the package, writes the code and runs it. If the first choice fails, it moves to the second one, the way a developer would.

Sandboxes. Each run happens in an isolated sandbox with a copy of the repository. Web access is on. Nothing touches a real service.

A simulated owner. The person asking is a model, not a human. Gemini 3.7 Flash plays the owner of the project. It sends the prompt, reads the agent's recommendation, checks it against written guidance for the sector, and only then asks for the code. This step matters. It separates "what the agent says" from "what the agent ships", and it stops an agent from installing something the owner would have refused.

A judge. A judge model reads every session and records which product the agent chose, which ones it only talked about, and what it installed. We review the judge's calls, and the run is published only after the recommendation, the code and the operating path pass that review.

Scale. 3,547 judged runs across 18 sectors, from deployment platforms to payments. 425 products in the registry. 585 hours of agent time. The median run takes 8 minutes.

What we found

Picks concentrate. The agents chose 180 different products across 3,547 runs, but inside each sector one or two favorites take most of the runs. Payments is the extreme case: Stripe wins 89% of runs. Bot protection goes to Cloudflare Turnstile at 66%, databases to Neon at 65%, cloud to AWS at 61%.

Four sectors are decided, eleven are open. We call a sector decided when one product wins at least half of the runs. Four sectors are. In the other eleven, the leader sits below 50% and the second product is close. Deployment is a good example: Vercel wins 38% and Render 37%.

In three sectors, the agent builds it itself. For agent frameworks, search and performance testing in continuous integration (CI), the most common outcome is no product at all. The agent writes the capability in-house. In agent frameworks that happens in 34% of runs, more than twice the best product. In search it is 24%, ahead of Postgres full-text search at 19%. In performance testing it is 43%. A vendor in these sectors competes with a few hundred lines of code the agent is happy to write.

Claude Code and Codex disagree in 8 of 18 sectors. Same repositories, same prompts, different winner. For sandboxes, Claude Code picks E2B and Codex picks Modal. For mail, Postmark against Resend. For deployment, Vercel against Render. For authentication, WorkOS AuthKit against Auth0. For product analytics, Claude Code builds it in-house while Codex picks PostHog. A product can lead one agent's board and be absent from the other.

Who asks changes the answer. Prompts written as a junior developer follow the sector leader in 56% of runs. Senior engineer prompts follow it in 39%, and enterprise prompts in 31%. Constraints and policy open the field. Enterprise repositories also pull the pick toward the cloud they already run on: Azure Database for PostgreSQL, Azure Blob Storage and Azure Communication Services appear on the boards only in runs where the repository is an Azure shop.

The repository decides more than the model's memory. In an earlier pilot, installing a product in the repository moved the agent's implementation from 0 of 15 runs to 11 of 15. When the task and the installed product contradict each other, the task wins. But reading the repository is a habit of the model, not of the tool: in the same pilots, GPT-5.6 Sol opened project files in 8 of 8 runs and followed what was installed, while GPT-5.6 Terra opened nothing and answered from memory.

Where the method is weak

We want the results used, so here are the limits.

• Two agents. Claude Code and Codex, plus a single Cursor run. Cursor, Gemini CLI and OpenCode boards are in progress.
• A moment in time. Models and agent versions are pinned to the run date. A model update can move a sector. That is why the run date is on every row.
• Our prompts. The 481 prompts are ours. They are frozen and public, so anyone can argue with them, but they are still a choice we made.
• A simulated owner. Gemini 3.7 Flash approves recommendations against written guidance. A human owner would sometimes decide differently.
• Uneven coverage. Sectors have between 60 and 480 runs. The thin boards are marked, and we do not draw conclusions from them.
• Judged, not asserted. A model reads the session and records the choice. We review those calls and publish the transcript, so you can check them.

How to use it

Every session is public. Open the leaderboards, pick a sector, and replay any run: the prompt, the searches, the commands and the code. If you build a developer tool, you can see whether agents name you, whether they pick you, and what they do when they do not.

We keep adding sectors and agents. If your category is missing, write to us and we will set up the runs.