Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents
Source: arXiv:2607.15263 · Published 2026-07-16 · By Paul Kassianik, Blaine Nelson, Yaron Singer
TL;DR
This paper presents a cost-aware evaluation framework for offensive and defensive AI security agents, arguing that standard peak-success metrics under generous budgets fail to capture operational realities. The authors compare language-model based security agents on offensive Cybench CTF challenges and defensive Splunk BOTS v1 incident investigations with a shared evaluation harness that measures success at fixed cost levels, decomposing cost into token inference spend and priced tool calls. Results reveal distinct scaling regimes: offensive task success improves predictably with more compute and command executions, with scaled open models nearing proprietary systems competitively on cost. Defensive SOC investigations, however, do not scale with raw budget in the same way; success depends more on judicious tool use, evidence navigation, and selective enrichment rather than brute inference power or tool volume. Public SOC benchmarks require contamination controls as no-tool baselines score surprisingly high, complicating capability claims. The authors emphasize economic efficiency and operational fit as necessary alongside success rates for real-world agent adoption.
Key findings
- On Cybench, GPT-5.5 achieves 94.1% success at $2.10 per challenge with $1.16 cost per solved challenge, outperforming other models.
- DeepSeek v4 Flash shows clear cost scaling on Cybench, improving from 76.1% success at $0.80 cap to 86.4% at $2.10 cap.
- Claude Opus 4.8 leads BOTS v1 defense with 93.9% points at $2.10 cost cap and best cost efficiency at $2.98 per 1000 points.
- On BOTS v1, increasing budget above $0.80 yields minimal score gain; for example, DeepSeek v4 Flash’s score increases only 0.9 percentage points when doubling budget from $2.10 to $4.20.
- High tool call volume in BOTS v1 does not correlate with better performance; Claude Opus 4.8 achieves top scores with fewer tool calls compared to DeepSeek runs.
- No-tool contamination controls reveal surprisingly high BOTS v1 scores (50-75%), indicating benchmark leakage and the need for decontamination controls.
- Offensive scoring declines sharply for models with policy refusals at test time (e.g., Fable 5 refuses all Cybench tasks yielding 0% success).
- Retrospective cost-capping and paired budget analyses show that offensive models benefit more from added budget than defensive models, emphasizing differing operational challenges.
Threat model
The threat model considers AI security agents as adversaries performing offensive or defensive tasks under fixed budget constraints. Offensive agents seek to solve CTF-style exploitation challenges using code execution and command tools, while defensive agents investigate telemetry and use priced tools for enrichment within SOC-style investigations. Agents cannot exceed set inference token or tool call budgets, do not control the environment, and their access to information is limited to available telemetry and permitted external enrichment APIs.
Methodology — deep read
The authors evaluate security AI agents using a unified framework (Inspect) to compare offensive CTF and defensive SOC tasks under explicit cost budgets, decomposed into inference spend and priced tool calls.
Threat Model: The adversary is simulated by AI security agents tackling red-team offensive CTF tasks (vulnerability exploitation, binary reverse engineering) on Cybench, and blue-team SOC defense tasks (investigations on telemetry using Splunk) on BOTS v1. The adversary models cannot exceed assigned budgets and do not have perfect knowledge; agents operate under budget constraints on token usage and tool call costs.
Data: Cybench hard subset consists of 39 sandboxed CTF-style challenges from multiple cyber competitions, each requiring flag capture with up to 3 submission attempts, scored by substring matches. BOTS v1 dataset includes a sequential 31-question Splunk investigation exercise with official scoring, hints with penalty costs, and telemetry logs. Public dataset, but with noted contamination risks.
Architecture/Agent: All runs use a ReAct-style agent with automatic context window compaction blending native provider compaction and summary fallback. Agents invoke bash, Python, submit, and in BOTS also Splunk search, event inspection, and enrichment APIs (Brave Search, WhoisXMLAPI, VirusTotal) with call limits and priced tool cost accounting. The agent is unaware of the cost budget but knows tool call limits.
Training Regime: The paper evaluates existing models (Claude Opus 4.8, GPT-5.5, GPT-5.6 variants, DeepSeek, Fable 5) via APIs; no training from scratch is reported. Each challenge is run for three epochs (independent runs), computing mean performance.
Evaluation Protocol: The key metric is success rate or points at fixed cost caps on cumulative token spending and tool call costs, analyzed both prospectively and retrospectively (replaying traces with lower cost caps). Refusals—runs where the agent declines to attempt a solution—are tracked and scored differently per benchmark. Contamination controls are applied on BOTS by running 'no-tool' baseline with zero tool access.
Reproducibility: An interactive website publishes results; most underlying data and code use public or shared benchmarks. Proprietary models are black-box API calls, so only evaluation harness and costs are transparent. Exact seeds and training details for base models are unspecified.
Example: On Cybench, the speed-cost tradeoff is shown by replaying DeepSeek v4 Flash runs capped at $0.80 and $2.10, observing a 10.3 percentage point improvement with higher budget due to more sandbox exploration and retries. On BOTS, Claude Opus 4.8 scores near the best within 603 tool calls, while DeepSeek uses >4,000 calls with lower scores, showing tool discipline matters more than volume.
Technical innovations
- Introducing a unified cost-aware evaluation framework that decomposes success by inference token spend and priced tool calls for security agents.
- Pairing offensive Cybench CTF benchmarks and defensive Splunk BOTS v1 datasets within a common agent harness to compare scaling regimes under fixed cost budgets.
- Applying retrospective budget caps to replay completed agent traces for paired controlled cost-performance analysis.
- Using no-tool contamination controls on public SOC benchmarks to expose data leakage and the inflation of absolute capability claims.
- Characterizing defense agent success as a function of disciplined tool use and evidence navigation rather than sheer inference budget or tool call volume.
Datasets
- Cybench hard-variant — 39 CTF-style sandboxed challenges — public benchmark (UK AI Security Institute)
- Splunk BOTS v1 — 31-question Splunk-based SOC investigation challenge — public telemetry dataset (Splunk)
Baselines vs proposed
- GPT-5.5 (Cybench): success rate = 94.1% vs DeepSeek v4 Flash (full budget): 86.4%
- DeepSeek v4 Flash (retrospective $0.80 cap): 76.1% vs full $2.10 cap: 86.4%
- Claude Opus 4.8 (BOTS v1): 93.9% at $2.10 cost vs GPT-5.6 Terra 92.1% at $2.10
- DeepSeek v4 Flash (BOTS v1): 73.0% at $2.10 cap vs $4.20 cap: 73.9%
- Claude Opus 4.8 $/1k points = $2.98 vs GPT-5.6 Sol $4.70
- BOTS v1 no-tools baseline Claude Opus 4.8: 74.8% vs full agent 93.9%
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.15263.
Fig 1: Resource scaling for five models per panel. Left: Cybench success versus a retrospective
Fig 2: Performance and policy refusals for the GPT-5.6-focused comparison. Hatched segments
Fig 3: BOTS v1 no-tools contamination probe. No-tools rows are single-epoch controls; full-agent
Fig 4: Cybench scaling curves for the previously audited model subset. Left: success versus
Fig 5: BOTS v1 scaling curves for the previously audited full-agent subset. Left: official BOTS
Fig 6: Tool-call scaling: success at cap x counts a sample-epoch only if it succeeded within x
Limitations
- Experiments are observational with retrospective budget capping rather than prospective randomized evaluations.
- BOTS v1 is an older public dataset with contamination risk; no-tools baseline exposes data leakage.
- Results rely on API-based evaluation of closed-weight models, limiting reproducibility and model introspection.
- Tool call limits and cost accounting exclude infrastructure, analyst time, and operational context costs.
- Evaluation is constrained to Cybench hard variant and BOTS v1 only; newer SOC benchmarks (BOTS v2/v3) are not covered.
- Policy refusals by models (e.g., Fable 5) complicate interpreting zero success as lack of capability.
Open questions / follow-ons
- How do newer and more complex SOC benchmarks (e.g., BOTS v2 and v3) impact cost-success scaling and tool discipline findings?
- Can agent cost accounting incorporate human analyst time and other infrastructure costs to better approximate operational efficiency?
- How do policy refusals and filtering affect the practical utility and evaluation fairness of language-model security agents?
- What architectural or prompting strategies improve defensive agent ability to prioritize high-value evidence and selective enrichment?
Why it matters for bot defense
Bot-defense and CAPTCHA practitioners can draw valuable insights from this paper’s framing of security-agent success as a function of economic and operational cost, not just peak accuracy. The demonstration that offensive capabilities scale with compute while defensive analyses depend more on selective, disciplined tool use highlights that bottlenecks in automated defense lie less in raw reasoning and more in workflow optimization and information triage. Evaluations in bot-defense should consider not only binary success/failure outcomes but also the efficiency of resource use, tool selections, and pathways that lead to attacks or defenses. Additionally, the contamination controls remind practitioners to carefully verify that publicly benchmarked agents do not benefit from training data leakage, a pitfall jeopardizing real-world robustness assessments. This cost-aware evaluation mindset can guide design and tuning of adaptive bot detectors and challenge-response systems that balance security efficacy against computational or user friction costs.
Cite
@article{arxiv2607_15263,
title={ Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents },
author={ Paul Kassianik and Blaine Nelson and Yaron Singer },
journal={arXiv preprint arXiv:2607.15263},
year={ 2026 },
url={https://arxiv.org/abs/2607.15263}
}