How coding agents choose tools

A developer types one sentence. A few minutes later a package is installed and the integration is written. Between those two moments there are about six steps, and each one is a place where a vendor wins or loses.

Here is the sequence, in the order it happens.

Step 1: the agent reads the repository

Before it does anything else, the agent looks around. It lists files. It opens the package manifest, the lock file, the README, the configuration, and any instruction file it finds such as AGENTS.md or CLAUDE.md.

This is where most of the decision gets made, and almost nobody optimises for it.

What the agent takes from this pass:

• The language and the framework. A TypeScript project with Next.js implies a different shortlist from a Django project.
• What is already installed. An existing client library for a competitor is close to decisive.
• The deployment target. A vercel.json file changes what "hosted" means.
• The constraints. A lint rule, a licence header, a compliance note in the README.
• Any explicit instruction. A line that says "use Postmark for email" is read and usually followed.

We measured the last two against each other. A single sentence in a repository file saying to use a particular email product was followed 16 times out of 18. Then we installed a competing email library into the same repository. The same sentence then lost two times out of three.

Step 2: the agent decides what the request means

The request is rarely a category name. Real requests look like "people keep signing up with fake emails" or "this page takes four seconds to load for our users in Australia".

The agent turns the symptom into a category. That translation is not neutral. "Fake signups" can become bot protection, email verification, rate limiting, or a CAPTCHA, and each of those has a different set of vendors.

This is why the same product can be strong in one wording of a request and absent in another. If your product solves a symptom, your documentation should use the symptom's words, not only the category's words.

Step 3: the agent produces candidates

Two sources.

Memory. Whatever the model learned during training. This favours products that were widely written about before the training cut-off, and it can be out of date. An agent will confidently install a version that no longer exists, or use an interface that was replaced.

Search. The agent runs its own queries. The queries are not the words the developer typed. They are much more specific, and they are often scoped to a site. Real examples from our runs:

site:meilisearch.com/docs typo tolerance ranking rules filterable attributes production self hosted
site:algolia.com pricing records search requests grow plan official
site:postgresql.org/docs/current pg_trgm GIN index similarity ILIKE official documentation

Notice what those queries want. Configuration details. Pricing units. Official documentation. Not a landing page, and not a blog post about why the product is great.

How often an agent searches at all varies enormously.

That difference is larger than almost anything a vendor can control. If your users mostly run Claude Code, web content is a weak lever and the repository is a strong one. If they run Codex, the opposite.

Step 4: the agent compares

The comparison is done against constraints found in step 1, not against a general idea of quality. The questions it is answering look like this:

• Does this work with the package manager already in use?
• Does it run in the environment this project deploys to?
• Does it need a service the project cannot add?
• Is there a version compatible with what is installed?
• Can I write working code for it from what I know or can read?

That last one is decisive and underrated. An agent prefers a product it can integrate correctly in one pass. A product with a five-line quickstart that works when pasted beats a better product with a quickstart that needs three unstated prerequisites.

Step 5: the agent proposes, and a person approves

In most real setups the agent says what it plans to do before it writes code. A person reads that and says go ahead, or asks a question.

This step is where hedging gets punished. An agent that answers "here are three good options, which do you prefer?" has not chosen anything. In our early tests, ten percent of file storage runs ended that way.

When we put a simulated project owner in the loop who insists on a specific answer, the leaders got smaller, not larger. In bot protection, one product took 83% of runs without an owner and 70% with one. In file storage, Amazon S3 went from 65% to 57% and a challenger went from zero wins to six.

Step 6: the agent writes the code

The final step is the real test. The agent installs the package, writes the calls, and often runs the code.

Products fail here for boring reasons. The package name does not match the product name. The quickstart assumes an environment variable that is never mentioned. The example uses an interface that changed two versions ago. The documentation is rendered by JavaScript and the agent fetched an empty page.

Every one of those is fixable in an afternoon, and every one of them costs installs.

What all of this adds up to

Four levers, in order of size.

The uncomfortable part is that the largest lever is the one you do not own. You cannot edit your customer's repository.

What you can do is get there first. Templates, starter kits, framework integrations, official examples, and platform defaults all put your product in the repository before the question is ever asked. That is why those investments pay far more in the agent era than they did before.

How consistent is any of this

Not very, and this matters for anyone trying to measure it.

The same agent, on the same repository, with the same request, disagrees with itself about a quarter of the time. So a single run tells you nothing. Neither does five. You need enough replicates that the noise averages out, which in practice means hundreds of sessions per category, not tens.

That is the main reason there was no public data on this before. One session costs between one dollar and fifteen in tokens and runs for about nine minutes. A category needs hundreds. Then something has to read every transcript and decide what was picked, and a person has to check that reader.

The published set behind this page is 5,292 judged sessions and 883 hours of agent time.