Benchmarking Autonomous Web App Pentest Solutions

This is Part 1 of our benchmark study, highlighting the overall empirical results.

In Part 2, we will dive beneath the surface to dissect individual findings, compare harness architectures, and break down the root causes behind the data.

TL;DR

Prioritizing deep contextual comprehension over scan speed in G4LAHAD, our purpose-built black/grey box harness for web application assessments, proved to be the right design decision:

1. High Impact Volume: Discovered 122 medium+ findings, all manually verified and reproduced by senior penetration testers, more than second and third place candidates combined.
2. Logic Dominance: Uncovered 8x more exclusive business logic and access control findings than all other candidates combined.
3. Blind Spot Detection: Identified 14x more exclusive findings than nearest candidate.
4. Engineered Depth: 1,004 test cases planned across 3 apps, outperforming the nearest candidate by 4x, surpassing human performance.
5. Operational Efficiency: 10x faster than humans with full proof of work and complete auditability.

The Verification Gap in Human-Dependent and Autonomous Pentesting

In cybersecurity, the golden rule is zero trust: never trust, always verify.

Yet human-dependent penetration testing is largely trusted without clear, repeatable proof. AI penetration testing faces the opposite challenge: bold claims are met with skepticism, creating a demand for clear, consistent benchmarks on real applications.

We built G4LAHAD as our autonomous web application penetration testing harness, purpose-built for black-box and grey box web application assessments, then evaluated against leading open-source solutions and agent harnesses on real-world applications.

Our goal isn't to declare a winner, but to share transparent data and lessons learned to help security teams make informed decisions and contribute to advancing the field.

Operational Reality: Why Black-Box and Grey Box Testing Matter

While white-box testing is comprehensive and fast, black- and grey box assessments mirror operational reality. The two approaches complement rather than duplicate each other: white-box examines the application from the inside out, while black- and grey box testing approaches it from an external adversary's viewpoint.

Furthermore, in practice, many clients simply do not own the application source code, rely on vendors who retain it, or are reluctant to share it.

We support all three modes across our portfolio, and even built a dedicated white-box harness, G4WAIN. However, for this benchmark, we deliberately chose to evaluate G4LAHAD in the dark: testing how far an autonomous offensive agent can go without the luxury of source code.

This operational caution also extends to data privacy. We recognize the risks of exposing sensitive application traffic to automated tools, which is why Zero Training Policy is a non-negotiable requirement for all AI providers used across 4rthur's engagements, with Zero Data Retention (ZDR) further required for sensitive or critical use cases.

The Benchmark Arena: Real-World, Multi-Role Web Applications

Synthetic CVE benchmarks evaluate known server or component vulnerabilities, not web applications where vulnerabilities live in business logic and multi-step workflows.

Furthermore, it's always thought that humans excel in context analysis and understanding. Whether autonomous pentest solutions can bridge that gap and truly grasp application context is the central question this benchmark explores.

We adopted ProjectDiscovery's open-source benchmark suite, featuring three multi-role applications built using modern AI coding tools without intentionally planted flaws, reflecting real-world production pipelines, where software is built fast with AI and vulnerabilities emerge organically from feature complexity:

• MedPortal: Healthcare portal handling Protected Health Information across patient, doctor, nurse, lab technician and admin roles
• VaultBank: Digital banking platform with customer, teller, branch manager, compliance officer and admin roles
• ClaimFlow: Insurance platform with role-based workflow transitions across policyholder, agent, adjuster, underwriter and admin roles

Candidates and Test Controls

We benchmarked four candidates under standardized conditions:

1. Strix OSS: Popular open source autonomous pentest agent with ~64k stars on GitHub.
2. Shannon OSS: Another popular open source AI web app pentest solution with ~49k stars on GitHub, built by Keygraph.
3. Claude Code: Anthropic's developer agent harness, to evaluate if general coding agents can replace specialized security tools.
4. G4LAHAD: Autonomous pentest harness purpose built for black and grey box web application assessment

Standardized Controls:

• Model: All candidates ran on the exact same model, GLM 5.3 Flash.
• Instruction: All tools received the same repository README with functional overviews, user roles, and valid credentials, without the repository and source-code architecture mirroring a standard grey box assessment.

Key Disclosure:

For full transparency, we explicitly benchmarked the open-source community editions: Strix OSS and Shannon OSS.

Regarding Shannon OSS specifically:

• White-Box Architecture: Shannon OSS requires full local source code access, as black-box and grey box modes are not supported in the open-source release.
• Vulnerability Focus: The open-source edition is engineered primarily for classical vulnerability classes: injection, cross-site scripting (XSS), SSRF, broken authentication and authorization flaws.
• Black-box, graph-based static analysis, SCA reachability, secrets detection and business logic testing are reserved for Keygraph Enterprise. Teams testing production systems should strongly consider Keygraph Enterprise.

Why include Shannon OSS in this comparison?

1. Answering the OSS Question: With massive community traction (~64k and ~49k GitHub stars), engineering leads frequently ask: why pay for commercial solutions when capable open-source alternatives exist? Benchmarking them provides a data-driven answer.
2. Measuring the Logic Gap: Evaluating Shannon OSS demonstrates how much of a web application's real-world attack surface is covered by classical vulnerability scanning versus dedicated business logic modeling.

We have immense respect for the teams at Keygraph and Strix. Our goal is not an asymmetrical comparison, but an honest examination of how different architectural approaches perform in practice.

Benchmark Result

Across all three applications, 369 vulnerabilities were validated through deduplication and human verification.

Informational and Low severity findings were recorded for completeness but excluded from primary metrics below to focus on impactful vulnerabilities.

Finding Count

G4LAHAD discovered 122 medium+ findings, more than second and third place candidates combined.

Findings by OWASP Top 10

Mapping findings to OWASP Top 10. More analysis on this later.

Exclusive Findings Count

Exclusive findings are vulnerabilities discovered by only one solution and missed by all other evaluated candidates.

Across all three applications, G4LAHAD identified 42 exclusive vulnerabilities, outperforming the nearest competitor by 14x.

Exclusive Findings by OWASP Top 10

OWASP Top 10 break down of exclusive findings.

Context is The Battlefield

G4LAHAD dedicates hours upfront solely to information gathering and contextual mapping. It reconstructs multi-role workflows, state transitions, and permission hierarchies before launching a single attack. It's an intentional architectural decision to prioritize deep contextual comprehension over scan speed.

The payoff is evident in the data: out of 46 exclusive findings across the entire benchmark, G4LAHAD uncovered 42 (91%) of them.

Why did other tools miss these vulnerabilities? The breakdown is telling: 38 (83%) of the 46 exclusive findings were either business logic or access control flaws, where the other harnesses combined found only 4, while G4LAHAD found 8x more (34).

Business logic flaws, such as balance minting, workflow bypasses, and off-ledger refunds, and access control breaches, such as role forgery and cross-tenant parameter tampering, both require deep contextual understanding of the application and its user roles.

This highlights the operational reality of web application security: vulnerabilities hide inside complex application logic, and discovering them requires an autonomous agent engineered to first understand context.

Engineered Depth: Test Case Volume and Surface Coverage

G4LAHAD's high finding count does not happen by accident, it is engineered. G4LAHAD builds an exhaustive, role-aware pre-attack execution plan that explores boundary states, unlisted parameters, and permission transitions across every authenticated session.

Surpassing Human Limits: 10x Speed with Full Proof of Work

Depth means little without speed. Speed means nothing without proof.

Executing 1,004 role-aware, multi-step test cases would take an experienced pentester roughly ten working days, during which cognitive fatigue inevitably sets in.

G4LAHAD completed the entire scope in 24 hours with relentless consistency: delivering 10x the speed, zero drop in test quality, and full proof of work recorded for every single test case, request trace, and outcome. No blind spots, and no "trust me" reports.

Frequently Asked Questions & Methodology Disclosures