Open seam
The thing a test repository is missing, which is what makes it a valid test. A repository that already has your category cannot measure choice.
An open seam is the capability a test repository is deliberately missing. A repository with no email sending has an open email seam, which is what makes it a valid place to measure how an agent chooses an email provider.
Why it decides whether a measurement is valid
If a repository already has a working email integration, and you ask an agent to add email sending, you are not measuring choice. You are measuring incumbency, and the answer is mostly already written.
Both are worth measuring. They are different experiments:
| Repository state | What the run measures |
|---|---|
| The seam is open | Choice. Which product an agent picks when the field is clear |
| A rival is installed | Incumbency. How hard it is to displace what is there |
| Your product is installed | Retention. Whether the agent keeps it or replaces it |
Mixing them in one number produces something that means nothing. Run them separately and report them separately.
What this costs in coverage
Being honest about the trade-off: insisting on an open seam reduces how many repositories a category can run on.
In our panel of 51 published repositories, bot protection ran on only five. The others had no public form worth protecting, so a bot protection request would have been artificial and the agent would have noticed.
Five valid repositories beats twenty invalid ones. A run in a repository where the request does not make sense produces a result, and the result is noise.
How each repository declares its seams
Every repository in a panel should state, in metadata the agent never sees, what it is missing. Then a category's runs only go to the repositories where its seam is open.
That metadata is stripped from the tarball the agent receives. If the agent can read the list of things the repository is missing, it has been handed the question.
The related rule: add a capability, never request a migration
A request that says "move us from Mailgun to something better" has an incumbent baked into it and a direction implied. That is a migration experiment.
A request that says "we need to send transactional email and we have none" is a choice experiment.
Both are legitimate. Keep them apart, because the first one's answer is largely determined by the words of the request and the second one's is determined by the repository and the agent.
Why the repository has to look real anyway
An open seam is necessary and not sufficient. The repository still has to convince an agent that pokes around.
In our panel each repository has a fake company name, a README written in that company's voice, a real lock file made by a real install, and a replayed git history with plausible authors and commit messages. There is a linter that checks every package hash against the live registry, because a hand-written lock file is the first thing an agent notices.
There is also one rule worth copying: a repository that mentions the company running the experiment, or another repository's fake company, fails the lint. The agent must never learn it is in a test.
Common questions
What is an open seam?
The capability a test repository is deliberately missing. A repository with no email sending has an open email seam, which makes it a valid place to measure how an agent chooses an email provider.
Why does an open seam matter?
Because a repository that already has a working implementation of your category measures incumbency, not choice. Both are worth measuring, but they are different experiments and mixing them makes the result meaningless.
How many repositories does a category need?
Enough with the right seam open. In our panel, bot protection ran on only five codebases, because the others had no public form worth protecting. Fewer valid repositories is better than more invalid ones.
Can the same repository have several open seams?
Yes, and most do. One repository can be missing authentication, email and payments at once, and serve as a valid test for all three, as long as the requests are run independently.
Where this comes from
Armature ran 5,292 judged sessions with Claude Code, Codex and Cursor inside 51 realistic codebases, and published every run. The numbers on this page come from that work.