Agent Security Needs Redefinition through a Holistic Framework
Source: arXiv:2607.22024 · Published 2026-07-24 · By Vincent Siu, Jingxuan He, Kyle Montgomery, Zhun Wang, Chenguang Wang, Dawn Song
TL;DR
This paper challenges the prevailing framing of agent security as primarily a question of action content—i.e., whether the commands agents perform look malicious. The authors argue this framing is fundamentally flawed because identical actions can be legitimate or malicious depending on contextual factors like who issued the command, the authorized objective (task), and information flow constraints. They propose a holistic formalization of agent security as a continuous evaluation of four core properties across the agent's execution trajectory: Source Authorization (who issued the command and authority), Task Alignment (whether the agent’s overall goal is authorized), Action Alignment (whether each executed action serves the authorized task), and Data Isolation (governing information flow across privilege boundaries). Empirical examples from benchmarks AgentDojo and WASP illustrate how current content-based evaluations conflate legitimate and unauthorized actions as indistinguishable, but the contextual framework separates these clearly. The paper reframes known attack classes like direct/indirect prompt injection and task drift in terms of violations of these properties, demonstrating missed detection gaps in today’s snapshot and content-only defenses. Contextual security also recasts existing defenses around the property they approximate and highlights critical missing defense approaches, especially for multi-step and cross-session memory poisoning attacks. This reconceptualization provides a principled foundation for agent security that moves beyond brittle content filters toward principled, continuous authorization verification.
Key findings
- All 45 indirect prompt injection tasks from AgentDojo and WASP have plausible legitimate scenarios, showing action content alone cannot distinguish attack from authorized behavior.
- Snapshot benchmarks reset context between tasks, making them structurally incapable of detecting Data Isolation violations like cross-session memory leaks.
- Indirect prompt injection is fundamentally a Source Authorization violation, as unauthorized external content issues commands the agent mistakenly treats as authentic.
- Direct prompt injection and jailbreaking occur when an authenticated user issues commands violating Task Alignment, not Source Authorization.
- Capability misuse—executing an unauthorized action under the authorized task—is an Action Alignment violation distinct from task drift.
- Memory poisoning cascades violations: initial Data Isolation violation enabling later Source Authorization violations downstream.
- PIGuard defense reports near random performance on benign prompts containing adversarial trigger words, explained as a failure of content-only defenses to approximate Source Authorization.
- Continuous evaluation of the four properties across an agent’s full execution trajectory is necessary to reveal temporal attack patterns invisible at single-step snapshot evaluations.
Threat model
The adversary can act as an unauthorized source injecting commands or content (e.g., malicious emails, webpage inputs) to an agent and/or as an authenticated user issuing commands conflicting with policy or permitted objectives. They cannot spoof authentication seamlessly or override enforced authorization checks on Source Authorization, Task Alignment, Action Alignment, or Data Isolation properties. The agent must continuously verify authorization context to avoid cascading violations enabled by memory poisoning or task drift.
Methodology — deep read
The paper redefines the threat model and agent security definition by explicit decomposition of contextual authorization into four properties: Source Authorization, Task Alignment, Action Alignment, and Data Isolation. The adversary model includes both external unauthorized sources injecting commands (e.g., malicious webpage content) and authenticated users issuing unauthorized objectives or actions (e.g., direct prompt injection, jailbreaking). Specific data examples are drawn from the public benchmarks AgentDojo and WASP, covering 45 injection tasks; the authors manually constructed plausible legitimate contexts for every task, demonstrating the structural ambiguity in content-only evaluations.
The authors provide conceptual formalism rather than new model architectures: the key innovation is the four-property contextual framework applied continuously over the agent’s execution trajectory rather than isolated snapshot checks. They trace how violations cascade through these properties in example attacks, e.g., memory poisoning introducing Data Isolation failures that later cause Source Authorization breaches.
No training or new machine learning model is proposed; rather, the analysis disentangles prior defenses (e.g., prompt injection filters, tool restrictions, memory isolation) by mapping which property they approximately address. Defenses like PIGuard, which rely on content classifiers, are shown structurally limited for Source Authorization violations.
Evaluation includes in-depth manual analysis of benchmark injection tasks (AgentDojo, WASP), illustrating that identical instructions can be benign or malicious depending on source and objective context. The paper also reviews existing defenses and benchmarks, demonstrating key gaps in capturing cross-session and multi-step attack vectors.
Reproducibility is conceptual; no code or formal toolchain is provided. However, comprehensive detailed discussion with concrete task examples and mappings to classical security authorization concepts grounds the framework’s applicability.
A concrete example end-to-end: The paper considers an agent processing a malicious email instructing deletion of stored user data. The same deletion command could be legitimate if issued by an authorized user. Snapshot content-based defenses fail to distinguish the cases. Under contextual security, the violation is identified as a Source Authorization breach (the email content came from an unauthorized source). This violation triggers blocking that is not achievable simply by content inspection of the command text.
Technical innovations
- Reframing agent security from action content detection to a contextual, continuous evaluation of four core authorization properties.
- Formally defining these properties—Source Authorization, Task Alignment, Action Alignment, and Data Isolation—as joint necessary criteria for trustable agent actions.
- Showing how well-studied attack classes map cleanly onto violations of these distinct properties, enabling targeted defense design.
- Demonstrating structural limitations of content-only and snapshot benchmarks in detecting authorization context violations, especially Data Isolation.
- Highlighting the need for continuous trajectory-based evaluation rather than isolated snapshot checks to expose cascading violations.
Datasets
- AgentDojo injection tasks — 45 tasks — public benchmark used for injection attack evaluation
- WASP injection tasks — subset of injection challenges from a public benchmark
Baselines vs proposed
- PIGuard defense on benign prompts with adversarial trigger words: near random classification, explained as expected due to content-only limitations vs. contextual authorization approach
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.22024.
Fig 2: The same prompt routed through the four properties under two different contexts. On the left, the deletion was requested by the
Limitations
- The work is conceptual and analytical; no new implemented system or defense framework beyond redefinition is provided.
- No empirical validation of continuous authorization frameworks or automated property checking is included—future work needed.
- Defense recommendations remain high level; specific designs for fine-grained Source Authorization or Data Isolation enforcement mechanisms are not detailed.
- The mapping to real-world deployed agents and scalability challenges of continuous evaluation over agent trajectories require investigation.
- The framework depends on precise formalization of authorization context, which may be challenging to specify or verify in complex systems.
- Indirect injection scenarios in benchmarks are manually paired with contexts; automated generation or adversarial evaluation of context is not addressed.
Open questions / follow-ons
- How to design scalable, automated runtime monitors or LLM-based judges that enforce the four authorization properties continuously over agent trajectories?
- What formal policy languages and enforcement mechanisms can precisely specify and verify Task Alignment and Action Alignment dynamically?
- How does contextual security integrate with multi-user, distributed agent systems where authorization contexts span organizations or domains?
- Can data provenance techniques be adapted or improved to enforce fine-grained Data Isolation across sessions for real-world deployed agents?
Why it matters for bot defense
For practitioners designing bot-defense or CAPTCHA mechanisms, this paper highlights a fundamental limitation in evaluating agent security solely via action content inspection. Attackers can embed functionally approved commands inside inputs where content looks legitimate but authority and context differ, enabling stealthy prompt injection or memory poisoning. Defense frameworks need to embed contextual verification—tracking command sources, authorized tasks, action-task consistency, and strict isolation of privileged data—rather than rely on static content filters. This suggests bot-defense tools should incorporate provenance-aware input validation, continuous authorization enforcement, and cross-session data management to detect and mitigate multi-step or contextually nuanced attacks that content-based heuristics inherently miss. It also advocates moving beyond snapshot evaluations toward continuous, trajectory-aware security assessments that model agent state evolution rather than isolated steps.
Cite
@article{arxiv2607_22024,
title={ Agent Security Needs Redefinition through a Holistic Framework },
author={ Vincent Siu and Jingxuan He and Kyle Montgomery and Zhun Wang and Chenguang Wang and Dawn Song },
journal={arXiv preprint arXiv:2607.22024},
year={ 2026 },
url={https://arxiv.org/abs/2607.22024}
}