How to measure install share

You can approximate this yourself in a weekend, and you should, before you buy anything or believe anyone.

What you are measuring

The output you want is not one number. It is a table:

An overall number with no splits will tell you something true and useless.

The weekend version

1. Build three repositories

Not empty folders. Real-looking projects in the stacks your users actually run.

Each one needs a real lock file made by a real install, a README, a plausible file structure, and a git history. If you can, use a real open source project of about the right shape and strip the parts that identify it.

The one rule that matters: the repository must have an open seam. Something your category is missing. If the repository already has an email client, an email request tests incumbency, not choice. Both are worth testing, separately.

2. Write ten requests in your users' words

This is where most home-made measurements go wrong.

Do not write "add bot protection". Write what your users actually type:

If the request names the category, you have skipped the step where the agent translates a symptom into a category, and that step is where a lot of products get lost.

Write a few in each voice: someone describing symptoms, someone naming the category with constraints, someone adding compliance requirements. Those produce different answers.

Freeze the wording before you run anything. Wording moves the result, so you want to measure that rather than be surprised by it.

3. Run them

Real command line tools, pinned to a version, in a throwaway sandbox. Claude Code and Codex at minimum, because they disagree.

Five runs per combination. Three repositories, ten requests, two agents, five runs is 300 sessions. At a dollar or two each in tokens and about nine minutes each, that is a few hundred dollars and a lot of parallel sandboxes.

Record the version of the tool before each run. When these tools update, their output changes, and results from two versions are not comparable.

4. Force a choice

An agent that answers "here are three good options, which do you prefer?" has chosen nothing, and you cannot count a shrug. In our early tests, ten percent of file storage runs ended that way.

Put a person in the loop, or a small fast model playing one. The flow: the agent recommends, the owner either asks one clarifying question or says go ahead, and only then does the agent write code.

Two rules for that owner, and they matter more than they sound:

• It may never name a product the agent did not raise.
• It may never ask for a list of options.

Without those rules you are measuring your own suggestions.

Worth knowing before you worry that this biases the result: in our data the owner made leaders smaller, not bigger. In bot protection, one product took 83% of runs without an owner and 70% with one. In file storage, Amazon S3 went from 65% to 57% and a challenger went from zero wins to six.

5. Decide what got picked

Read the sessions. If you have 300, read them with a model and check a sample by hand.

If you use a model as the judge, use one from a family that does not build any of the agents you tested. A Claude judge recognises Claude's prose. A GPT judge recognises GPT's. We measured this: with clear rules, three copies of our judge agreed with each other 97.9% of the time. Without them, 84%.

Rules that made the difference:

• One pick per product. Never "Prometheus slash Grafana".
• A product named only inside a shell command still counts.
• A proposed default that the agent then builds on is a pick, at low confidence.
• A vendor-neutral standard is never the winner. OpenTelemetry was mentioned in 382 runs and won zero, by rule, because it is the wiring and not the destination.

6. Read the losses

This is the part that pays for the whole exercise.

Take the sessions where you lost and read what happened. In our data the reasons cluster into a short list, and most of them are defects rather than preferences:

• The quickstart did not run, so the agent abandoned the integration part way through.
• The documentation described an interface that had changed.
• The package name did not match the product name and the install went wrong.
• The agent could not find an answer to a specific production question.
• The fit was genuinely wrong for that repository, which is fine and worth knowing.

Four of those five are fixable in days.

The traps

Deciding which runs count after you see them. Fix the rule at launch: the first N valid runs in an order set in advance, extras cancelled. Otherwise you will keep your good runs without meaning to.

Retrying failures. A run that failed is a result. Retrying until it works measures your patience.

Counting a truncated log as a success. An exit code of zero with an incomplete event stream is a crash. Require the tool's own final event before you count anything.

Testing in an empty folder. This is the big one. An empty folder measures the model's memory. A repository measures the market.

One run per cell. The same agent on the same repository disagrees with itself about a quarter of the time. Five runs minimum, more if the numbers are close.

Letting the repository name you. If your product appears anywhere in the repository, including in a comment or a stray dependency, you have planted the answer. Lint for it.

What it costs

Honest numbers, from doing this at scale.

One session is a dollar to fifteen dollars in tokens and about nine minutes. A category needs hundreds of sessions to be stable, not tens. Adding an owner in the loop multiplied our bill by roughly four, because every run gained a recommendation turn, up to two questions, and then the implementation.

The published set behind these pages is 883 hours of agent time, and the boards we threw away are not in that number.

That cost is the reason this data did not exist before. It is also why the weekend version is worth doing first: 300 runs will tell you whether the full version is worth paying for.