Agent leaderboards / All sectors / Security testing

Security testing: which security tools coding agents choose

ZAP leads. Asking for an AI pen tester changes the choice.

531 runs7 apps4 agents1 personaupdated 2026-11-13

The interactive board, open on security testing. Open it full page · this page as Markdown

Read this leaderboard as textrankings, key learnings, method

Key learnings

We asked coding agents to test the security of seven applications, from a first automated check to a full pen test.

ZAP is the default against a running app

ZAP is chosen in 24% of all runs and in 54% of the access checks between users and tenants.

Source-only reviews go to Semgrep

Semgrep is chosen in 53% of source-only reviews and in 93% of runs on the Python settlement diff. SonarQube follows at 22%.

An AI pen-test ask moves away from ZAP and Burp

A plain pen-test ask goes to ZAP or Burp Suite in 83% of runs. When the ask says AI pen tester, they fall to 11%, and Strix and Escape are each chosen in 23% of runs.

Explore every run in the interactive board

The ranking531 runs

ProductWinsShare
1 ZAPzaproxy.org 126 24%
2 Semgrepsemgrep.dev 86 16%
3 SonarQubesonarsource.com 70 13%
4 Burp Suiteportswigger.net 34 6%
5 GitHub Advanced Securitygithub.com 28 5%
6 Strixusestrix.com 23 4%
7 Trivytrivy.dev 17 3%
8 StackHawkstackhawk.com 13 2%
9 Escapeescape.tech 10 2%
10 Hurlhurl.dev 7 1%
11 Shannongithub.com 7 1%
12 OWASP Dependency-Checkowasp.org 6 1%
13 govulncheckgo.dev 6 1%
14 Burp Suite + ZAP 6 1%
15 XBOWxbow.com 6 1%
16 OSV-Scannergoogle.github.io/osv-scanner 5 1%
17 Banditbandit.readthedocs.io 4 1%
18 GitLab Securitygitlab.com 4 1%
19 Sonatype Lifecyclesonatype.com 3 1%
20 Cobaltcobalt.io 3 1%
21 Snyksnyk.io 3 1%
22 Aikido Securityaikido.dev 3 1%
23 Keygraphkeygraph.io 2 0%
24 CodeQLgithub.com 2 0%
25 Contrast Assesscontrastsecurity.com 2 0%
26 Datadog Code Securitydatadoghq.com 2 0%
27 Codex Securityopenai.com 2 0%
28 Playwright Testplaywright.dev 2 0%
29 Semgrep + ZAP 2 0%
30 SonarQube + Trivy 2 0%
31 Newmanpostman.com 2 0%
32 42Crunch42crunch.com 2 0%
33 Aptori Siftaptori.com 2 0%
34 Claude Securityanthropic.com 2 0%
35 Bright Securitybrightsec.com 2 0%
36 Promptfoopromptfoo.dev 1 0%
37 Burp Suite + PentestGPT 1 0%
38 Vaadata 1 0%
39 Newman + Postman 1 0%
40 Claude Security + govulncheck 1 0%
41 Dependency-Track + SonarQube 1 0%
42 GitHub Agentic Workflowsgithub.com 1 0%
43 Checkmarx Onecheckmarx.com 1 0%
44 govulncheck + Semgrep 1 0%
45 Burp Suite + Nessus + punch-q + ZAP 1 0%
46 SonarQube + Trivy + ZAP 1 0%
47 Trivy + ZAP 1 0%
48 Burp Suite + Invicti 1 0%
49 Built in-house + ZAP 1 0%
50 Psalmpsalm.dev 1 0%
51 Gitleaks + OWASP Dependency-Check + Semgrep + ZAP 1 0%
52 GitLab Security + Kubesec + Semgrep 1 0%
53 GitLab Security + Semgrep 1 0%
54 Invictiinvicti.com 1 0%
55 Find Security Bugs + SpotBugs 1 0%
56 Equixly 1 0%
57 Nucleiprojectdiscovery.io 1 0%
58 Bishop Foxbishopfox.com 1 0%
59 OWASP Dependency-Check + SonarQube 1 0%
60 Claude Code Security Reviewanthropic.com 1 0%
61 OWASP Coraza + OWASP Core Rule Set 1 0%
62 Psalm + Trivy 1 0%
63 Barrion 1 0%
64 Red Hat Advanced Cluster Security for Kubernetesredhat.com 1 0%
65 Falcofalco.org 1 0%
66 Semgrep + Trivy + ZAP 1 0%
67 Semgrep + Trivy 1 0%
68 Pentest-Tools.com 1 0%

By agent, by persona, by wording

By agent

Claude Code187 runsZAP · 38then Semgrep · 29
Grok Build CLI · Grok 4.7116 runsZAP · 21then Semgrep · 19
Codex · GPT-6 Sol116 runsZAP · 28then Semgrep · 22
Muse Code · Muse Spark 1.3112 runsZAP · 39then SonarQube · 22

By persona

Enterprise team531 runsZAP · 126then Semgrep · 86

By what the ask stressed

The plain ask526 runsZAP · 123then Semgrep · 86

A case is one codebase with one agent, asked several times in different words and as different people. 28 of 28 cases did not hold to a single security tool.

How this was measured

Every number on this page comes from a controlled experiment. We took 7 small applications, asked 4 coding agents (Claude Code, Grok Build CLI (Grok 4.7), Codex (GPT-6 Sol), Muse Code (Muse Spark 1.3)) to test the security of each of them, in several wordings and as an enterprise team, and let the agent choose the product. Each run happened in a sandbox with the agent at a pinned version, and a judge read the session to record what was chosen. That is 531 runs. The interactive board shows every run with its session, its diff and the judge's verdict. A simulated user stood in for the owner of the codebase: it read the agent's plan and had to approve it before any code was written; it sent the agent back at least once in 63 runs. Read the methodology and the publications.

If you sell in this sector: what these numbers mean for a vendor.

Open the interactive boardThis page as Markdown