Judge model

At any real scale there is no alternative. Reading 5,292 sessions by hand is not a project anyone finishes.

Pick a family that is not competing

The single most important choice, and the one people skip.

A model recognises its own family's prose. A Claude judge can tell when a session was produced by Claude. A GPT judge can tell when a session was produced by GPT. That is a bias you cannot easily detect afterwards and can trivially avoid up front.

So: if you are testing Claude Code, Codex and Cursor, use a judge from a family that builds none of them. We used Gemini, not because it reads best, but because it has no stake.

Make it blind

The judge should see:

• A redacted transcript
• The diff
• The file tree
• A frozen list of candidate products

It should not see:

• Which agent produced the session
• Which model was behind it
• That any of this is a test

Relabel the simulated owner's messages as coming from "user". If a tool name or model name leaks into the evidence, flag it as a leak rather than ignoring it.

The rules matter more than the model

We measured this. Three copies of the same judge, on the same sessions:

Thirteen points, from writing the rules down. The rules that did most of the work:

One pick per product. Never "Prometheus slash Grafana". If the session used two products together, that is a combination and it is recorded as one outcome, not two wins.

A product named only inside a shell command still counts. An agent that runs npm install acme chose Acme, whatever the prose said.

A proposed default that the agent then builds on is a pick, recorded at low confidence.

A vendor-neutral standard is never the winner. OpenTelemetry was mentioned in 382 sessions and won zero, by rule, because it is the wiring rather than the destination. Without that rule it would have topped observability and told you nothing.

Record a role for every product, not only the winner: primary pick, alternative, mention, rejected. That is what produces the consideration rate.

Record evidence. A file path and line, or a transcript location, for every verdict. A verdict with no pointer cannot be checked.

Publish the judge's error rate

A judge undercounts. It misses names.

We track that as the judge gap: products the transcript mentions that the judge did not list. In observability, 155 of 360 sessions had at least one.

Publishing that number is better than pretending it is zero. It tells a reader how much to trust each figure, and it is the difference between a measurement and a marketing claim.

Have a person check it

Sample the verdicts and read them yourself. In our case, each version of the judge came from finding it wrong on a real case. Version nine exists because versions one through eight were each broken in a specific, discoverable way.