Playbooks

How to get picked for security testing by coding agents

ZAP took 24% of 531 judged security testing sessions. What the numbers say a vendor in this category should do.

Published October 14, 2026 Updated November 13, 2026 Read as Markdown

If you sell security testing tools, this page is the part of the market no dashboard shows you: what a coding agent does when a developer asks for security testing and never compares vendors.

The numbers come from 531 judged sessions with Claude Code, Codex, Grok Build CLI and Muse Code, spread across 7 realistic codebases, with every session read by a judge.

What coding agents choose for security testing

Across 531 judged sessions, ZAP was chosen most often, in 24% of runs. Semgrep was second with 16%.

#ProductRuns wonShare
1ZAP12624%
2Semgrep8616%
3SonarQube7013%
4Burp Suite346%
5GitHub Advanced Security285%
6Strix234%
7Trivy173%
8StackHawk132%
9Escape102%
10Hurl71%

Full board, every run replayable: the security testing leaderboard.

What the shape of this category means

The leader takes only 24% of runs. This category is genuinely open and the ordering can be moved.

With the top product at 24%, security testing is decided in the moment, from what the agent reads and what it finds in the repository. Nothing is locked in, which is the best situation a vendor can be in and the one where the work pays fastest.

The order here is set by the quality of what an agent can read and by whether your product is already present in the codebase. Both are things you can change.

The agents do not agree with each other

In this category Claude Code and Codex put ZAP first, which is less common than it sounds: across the 31 categories we measured, Claude Code and Codex disagreed on the leader in 16 of them.

Grok Build CLI (116 sessions) and Muse Code (112 sessions) join this table with fewer runs, so read their rows as indicative.

AgentRunsPicked most often
Claude Code187ZAP (38)
Codex116ZAP (28)
Grok Build CLI116ZAP (21)
Muse Code112ZAP (39)

Even where they agree, they get there differently. In this category Codex ran a web search in 100% of its sessions and Claude Code in 44%, so what you publish reaches nearly all of Codex's security testing sessions and more than a third of Claude Code's.

What you are really competing against

In this category agents never chose to build it themselves. All but 2 sessions ended with a product. That is good news: you are in a straight vendor comparison, and the levers that work are the ones you control.

Considered, and never chosen

Because the judge records every product an agent raised and not only the one it picked, this board also shows who kept reaching the shortlist and losing. In security testing the clearest case is SonarQube Cloud: on the table in 39 sessions, chosen in none.

ProductRaised inChosen in
SonarQube Cloud39 sessions0
Horizon3.ai NodeZero27 sessions0
Pentera23 sessions0

Being rejected is a better position than being unknown, and a cheaper one to fix. The product is already in the agent's head and on the list. Whatever ended those 89 sessions is recorded in each transcript, one reason at a time.

What to do about it in security testing

  1. Skip the build-versus-buy argument. No security testing session in this experiment ended with the agent writing its own implementation. All but 2 adopted a product, so the whole contest is against the other names in the table above.
  1. Treat the ordering as movable. The leader holds 24%, so agents are deliberating rather than defaulting, and the inputs they use can change the answer. This is the most winnable shape a category comes in.

The work that applies to every category rather than to this one is written up separately: audit your documentation, write a quickstart an agent can follow, and how to measure install share.

Every security testing tool on this board

One page per product, with its install share, the per-agent split, and how often it was raised without being chosen.

Where these numbers come from

531 judged sessions in security testing across 7 codebases, part of a published set of 15,379. Real coding agents at pinned versions, in sandboxes, inside realistic codebases, with a simulated project owner in the loop and a blind judge on every session. The full method is on one page: how we measured this.

Every security testing run can be replayed on the board.

<!-- generated by scripts/write-data-pages.mjs -->

Common questions

How many codebases is this based on?

531 judged sessions across 7 realistic codebases. A category only runs on repositories where its seam is open, so coverage differs: some categories ran on more than ten codebases and some on two.

What security testing tool do coding agents choose?

Across 531 judged sessions, ZAP was chosen most often, in 24% of runs. Semgrep was second with 16%. Every agent put ZAP first.

Do Claude Code and Codex pick the same security testing tool?

Yes. Claude Code and Codex both put ZAP first in this category, which is less common than it sounds: they disagree on the leader in 16 of the 31 categories we measured.

How often do agents build security testing themselves instead of installing something?

Never, in this category. No session ended with the agent writing the code itself. All but 2 adopted a product; the rest used built-in tools or made no choice.

How can a vendor improve its position here?

Make the quickstart run when pasted, state the current version on the documentation page, use one name across product, package and import, write pages for the symptoms users describe rather than only the category name, and get into the repository through templates and framework integrations.

Which security testing tools do agents consider but never choose?

SonarQube Cloud (raised in 39 sessions, chosen in none), Horizon3.ai NodeZero (raised in 27 sessions, chosen in none), Pentera (raised in 23 sessions, chosen in none). Being considered and not chosen is a different problem from being unknown, and it is usually fixable.

Where this comes from

Armature ran 15,379 judged sessions with Claude Code, Codex, Cursor, Grok Build CLI and Muse Code inside 110 realistic codebases, and published every run. The numbers on this page come from that work.

Read next

All library pages