Agent leaderboards / All sectors / Security testing
Security testing: which security tools coding agents choose
ZAP leads. Asking for an AI pen tester changes the choice.
Read this leaderboard as textrankings, key learnings, method
Key learnings
We asked coding agents to test the security of seven applications, from a first automated check to a full pen test.
ZAP is the default against a running app
ZAP is chosen in 24% of all runs and in 54% of the access checks between users and tenants.
Source-only reviews go to Semgrep
Semgrep is chosen in 53% of source-only reviews and in 93% of runs on the Python settlement diff. SonarQube follows at 22%.
An AI pen-test ask moves away from ZAP and Burp
A plain pen-test ask goes to ZAP or Burp Suite in 83% of runs. When the ask says AI pen tester, they fall to 11%, and Strix and Escape are each chosen in 23% of runs.
The ranking531 runs
| Product | Wins | Share | ||
|---|---|---|---|---|
| 1 | ZAPzaproxy.org | 126 | 24% | |
| 2 | Semgrepsemgrep.dev | 86 | 16% | |
| 3 | SonarQubesonarsource.com | 70 | 13% | |
| 4 | Burp Suiteportswigger.net | 34 | 6% | |
| 5 | GitHub Advanced Securitygithub.com | 28 | 5% | |
| 6 | Strixusestrix.com | 23 | 4% | |
| 7 | Trivytrivy.dev | 17 | 3% | |
| 8 | StackHawkstackhawk.com | 13 | 2% | |
| 9 | Escapeescape.tech | 10 | 2% | |
| 10 | Hurlhurl.dev | 7 | 1% | |
| 11 | Shannongithub.com | 7 | 1% | |
| 12 | OWASP Dependency-Checkowasp.org | 6 | 1% | |
| 13 | govulncheckgo.dev | 6 | 1% | |
| 14 | Burp Suite + ZAP | 6 | 1% | |
| 15 | XBOWxbow.com | 6 | 1% | |
| 16 | OSV-Scannergoogle.github.io/osv-scanner | 5 | 1% | |
| 17 | Banditbandit.readthedocs.io | 4 | 1% | |
| 18 | GitLab Securitygitlab.com | 4 | 1% | |
| 19 | Sonatype Lifecyclesonatype.com | 3 | 1% | |
| 20 | Cobaltcobalt.io | 3 | 1% | |
| 21 | Snyksnyk.io | 3 | 1% | |
| 22 | Aikido Securityaikido.dev | 3 | 1% | |
| 23 | Keygraphkeygraph.io | 2 | 0% | |
| 24 | CodeQLgithub.com | 2 | 0% | |
| 25 | Contrast Assesscontrastsecurity.com | 2 | 0% | |
| 26 | Datadog Code Securitydatadoghq.com | 2 | 0% | |
| 27 | Codex Securityopenai.com | 2 | 0% | |
| 28 | Playwright Testplaywright.dev | 2 | 0% | |
| 29 | Semgrep + ZAP | 2 | 0% | |
| 30 | SonarQube + Trivy | 2 | 0% | |
| 31 | Newmanpostman.com | 2 | 0% | |
| 32 | 42Crunch42crunch.com | 2 | 0% | |
| 33 | Aptori Siftaptori.com | 2 | 0% | |
| 34 | Claude Securityanthropic.com | 2 | 0% | |
| 35 | Bright Securitybrightsec.com | 2 | 0% | |
| 36 | Promptfoopromptfoo.dev | 1 | 0% | |
| 37 | Burp Suite + PentestGPT | 1 | 0% | |
| 38 | Vaadata | 1 | 0% | |
| 39 | Newman + Postman | 1 | 0% | |
| 40 | Claude Security + govulncheck | 1 | 0% | |
| 41 | Dependency-Track + SonarQube | 1 | 0% | |
| 42 | GitHub Agentic Workflowsgithub.com | 1 | 0% | |
| 43 | Checkmarx Onecheckmarx.com | 1 | 0% | |
| 44 | govulncheck + Semgrep | 1 | 0% | |
| 45 | Burp Suite + Nessus + punch-q + ZAP | 1 | 0% | |
| 46 | SonarQube + Trivy + ZAP | 1 | 0% | |
| 47 | Trivy + ZAP | 1 | 0% | |
| 48 | Burp Suite + Invicti | 1 | 0% | |
| 49 | Built in-house + ZAP | 1 | 0% | |
| 50 | Psalmpsalm.dev | 1 | 0% | |
| 51 | Gitleaks + OWASP Dependency-Check + Semgrep + ZAP | 1 | 0% | |
| 52 | GitLab Security + Kubesec + Semgrep | 1 | 0% | |
| 53 | GitLab Security + Semgrep | 1 | 0% | |
| 54 | Invictiinvicti.com | 1 | 0% | |
| 55 | Find Security Bugs + SpotBugs | 1 | 0% | |
| 56 | Equixly | 1 | 0% | |
| 57 | Nucleiprojectdiscovery.io | 1 | 0% | |
| 58 | Bishop Foxbishopfox.com | 1 | 0% | |
| 59 | OWASP Dependency-Check + SonarQube | 1 | 0% | |
| 60 | Claude Code Security Reviewanthropic.com | 1 | 0% | |
| 61 | OWASP Coraza + OWASP Core Rule Set | 1 | 0% | |
| 62 | Psalm + Trivy | 1 | 0% | |
| 63 | Barrion | 1 | 0% | |
| 64 | Red Hat Advanced Cluster Security for Kubernetesredhat.com | 1 | 0% | |
| 65 | Falcofalco.org | 1 | 0% | |
| 66 | Semgrep + Trivy + ZAP | 1 | 0% | |
| 67 | Semgrep + Trivy | 1 | 0% | |
| 68 | Pentest-Tools.com | 1 | 0% |
By agent, by persona, by wording
By agent
| Claude Code187 runs | ZAP · 38then Semgrep · 29 |
| Grok Build CLI · Grok 4.7116 runs | ZAP · 21then Semgrep · 19 |
| Codex · GPT-6 Sol116 runs | ZAP · 28then Semgrep · 22 |
| Muse Code · Muse Spark 1.3112 runs | ZAP · 39then SonarQube · 22 |
By persona
| Enterprise team531 runs | ZAP · 126then Semgrep · 86 |
By what the ask stressed
| The plain ask526 runs | ZAP · 123then Semgrep · 86 |
A case is one codebase with one agent, asked several times in different words and as different people. 28 of 28 cases did not hold to a single security tool.
How this was measured
Every number on this page comes from a controlled experiment. We took 7 small applications, asked 4 coding agents (Claude Code, Grok Build CLI (Grok 4.7), Codex (GPT-6 Sol), Muse Code (Muse Spark 1.3)) to test the security of each of them, in several wordings and as an enterprise team, and let the agent choose the product. Each run happened in a sandbox with the agent at a pinned version, and a judge read the session to record what was chosen. That is 531 runs. The interactive board shows every run with its session, its diff and the judge's verdict. A simulated user stood in for the owner of the codebase: it read the agent's plan and had to approve it before any code was written; it sent the agent back at least once in 63 runs. Read the methodology and the publications.
If you sell in this sector: what these numbers mean for a vendor.
Open the interactive boardThis page as MarkdownOther sectors
- Agent sandboxes
- Observability
- AI SRE
- Payments
- Deploy
- Auth
- Email providers
- Product analytics
- Databases
- File storage
- LLM evals & observability
- Voice Agents
- Serverless functions
- Cloud
- AI gateway
- Bot protection
- Search
- Agent frameworks
- Performance in CI
- Document processing & OCR
- Security platforms
- Usage-based billing
- Internationalization
- Maps
- Message queues
- AI search
- Code review
- E-signature
- In-app chat & calls
- Vector search