Evaluating autonomous reasoning, security tool execution, evidence collection, and finding validation in real security workflows
We’ve been building Rakshak V2 for quite some time, and during the development process, one fundamental question kept coming up:
Instead of just testing individual isolated features or simple synthetic prompts, we put Rakshak V2 through a structured benchmark to see how the complete system performs under realistic conditions.
The benchmark covered different stages of the autonomous security pipeline — from reconnaissance and security tool execution to analysis, evidence collection, finding validation, and automated report generation.
And honestly, the most useful part wasn’t just looking at the final scores. It was seeing where the system worked well, where it made mistakes, and which specific architectural components required further engineering effort.
The core objective was to test the platform as an integrated end-to-end unit, rather than treating an LLM or an individual security CLI tool as the product by itself. We evaluated how Rakshak handles:
Discovering target assets, domain structures, open ports, and surface attack vectors autonomously.
Invoking scanners, parsers, and penetration testing tools with contextual parameter selection.
Enforcing out-of-scope boundaries and strict target constraints during automated tasks.
Correlating multi-source findings to determine actual security exposure and risk severity.
Capturing non-repudiable logs, raw HTTP responses, network traces, and command outputs.
Eliminating false positives by verifying potential flaws with deterministic validation checks.
Evaluating model decision loops, adaptivity upon failure, and multi-step plan adjustments.
Compiling technical data, remediation guidelines, and executive summaries automatically.
| Pipeline Stage | Evaluation Focus | Key Metric Target |
|---|---|---|
| Recon & Scope | Asset identification accuracy & strict scope boundary enforcement | Zero Out-of-Scope Execution |
| Tool Orchestration | Correct CLI tool selection, dynamic flag tuning, & error recovery | Tool Command Success Rate |
| Analysis & Validation | False positive reduction through multi-stage deterministic checks | Verifiable Finding Ratio |
| Evidence & Reporting | Complete proof of concept log generation & structured executive summaries | Audit-ready Technical Reports |
Building an autonomous security system is significantly harder than simply connecting an AI model to security tools.
While modern foundation models excel at high-level reasoning and planning, the surrounding software architecture must guarantee that reasoning reliably translates into correct, deterministic, and verifiable actions.
That’s exactly why we prioritized running a rigorous benchmark at this stage of development. We would rather identify edge cases and failure modes ourselves during internal stress testing than discover them later in production deployments.
We don’t view this benchmark as a final static grade for Rakshak V2. For our engineering team, it represents a clear, objective snapshot of where the platform stands today and highlights the immediate path forward.
Rakshak V2 is under continuous development, and the empirical data gathered from this benchmark will directly shape upcoming platform iterations and agent control loops.
We’ve published the full technical benchmark report separately, covering our evaluation methodology, testing criteria, and comprehensive performance metrics.
Read The Full Benchmark Report →We will continue to share more of the underlying engineering, architectural patterns, and security research behind Rakshak as development progresses.
— Rudraksh AGI