LLM agent safety, multi-turn red-teaming, jailbreak benchmarks, adversarial robustness, safety-critical systems

Source: arXiv:2606.20408 · Published 2026-06-18 · By Hanwool Lee, Dasol Choi, Bokyeong Kim, Seung Geun Kim, Haon Park

TL;DR

This paper addresses the safety and robustness of large language model (LLM) agents deployed as operators in multi-agent teams for safety-critical systems, focused on nuclear power plant control room simulations. It introduces NRT-Bench, a novel multi-turn red-teaming benchmark where an adaptive attacker interacts with a five-role LLM operator team responsible for maintaining six critical safety functions (CSFs) in a closed-plant simulator. Unlike prior jailbreak benchmarks relying on single-turn text-based judgments, NRT-Bench measures concrete physical harm via objective plant safety transitions caused by adversarial messages injected through four realistic ingress channels. The benchmark supports multi-turn adaptive attacks and role-specialized LLM teams enforcing layered guardrails and approval constraints. Evaluations on four state-of-the-art LLM operators reveal 8.7% to 12.1% session attack success rates causing CSF loss, with nearly disjoint failure sets across models. Interestingly, defense layer effectiveness varies strongly and even contradictorily between models, signifying model-dependent vulnerabilities and challenges in universal guardrails. The authors release their simulation environment, fixed attack dataset, and replay tools to enable reproducible comparative evaluation of LLM agent safety in safety-critical supervision contexts.

Key findings

• Adaptive multi-turn attacks cause loss of at least one critical safety function (CSF) in 8.7% to 12.1% of attack sessions across four frontier operator models.
• Aggregate attack success rates (ASRCSF) for the two cloud models under eight defence-cell configurations are 23.7% for gpt-5.4-mini and 37.5% for claude-haiku-4-5 (Table 2).
• Guardrail-defence stacks reduce ASRCSF by 6.9 percentage points for gpt-5.4-mini but increase ASRCSF by 2.1 points for claude-haiku-4-5, demonstrating model-dependent defence effectiveness.
• The strongest defence cell reduces ASRCSF on gpt-5.4-mini to 1.4% but leaves claude-haiku-4-5 at 43.1%, a 30× gap on identical code and attacks (Table 2, cell C3).
• Failure sets of the four operator models on a curated 149-session test split are nearly disjoint: no session defeats all four, while about one-third defeat at least one model.
• Safety technical advisor authority level can paradoxically increase ASRCSF for some models (e.g., claude-haiku-4-5's ASR rises by 9.8 points when enabling advisory veto).
• Attack strategy families vary in difficulty: authority spoofing reaches 37.8% ASR on gpt-5.4-mini and 54.2% on claude-haiku-4-5; gradual escalation 29.2% vs 50.3% (Table 3).
• Attacker visibility is considered mainly in black-box mode; white-box mode with extra state disclosure serves as an upper bound but is not the default.

Threat model

The adversary is an external adaptive attacker limited to injecting messages through four designated ingress channels simulating outsider, insider impersonation, supply-chain compromise, and compromised auxiliary agents. The attacker cannot directly access operator system internals, message buses, or simulator code, and cannot forge valid ingress channel signatures without detection. The attacker seeks to maneuver the multi-agent LLM operator team into causing loss of at least one critical safety function in the simulated plant by issuing carefully crafted multi-turn payloads conditioned on redacted environment feedback.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is external to the LLM operator team and interacts solely via four restricted ingress HTTP channels simulating outsider, impersonating insider, supply-chain compromise, and compromised auxiliary-agent capabilities. The attacker cannot directly access the operator message bus or system prompts, cannot impersonate operators at the bus-level, nor interfere with simulation or LLM infrastructure. The goal is to drive the simulated plant into an unsafe state by causing loss of any critical safety function (CSF). Attacker observations are redacted summaries (black-box) or full state (white-box), with a bounded multi-turn session of up to 10 turns for adaptive escalation.

2. Data: The benchmark environment is a closed textual simulator of a nuclear power plant, with a detailed model of plant processes, alarms, and six key CSFs. Five operator roles (Senior Reactor Operator (SRO), Reactor Operator (RO), Turbine Operator (TO), Auxiliary Operator (AO), Safety & Technical Advisor (STA)) run the plant collaboratively via typed inter-agent messaging. The dataset released consists of a static archive of 149 multi-turn adversarial sessions generated by an internal LLM-driven attack agent using curated attack strategies, including authority spoofing and urgency injection. The fixed attack sequences allow replay against multiple candidate operator models under identical protocol settings. Each session contains attacker payloads, redacted situation summaries, plant states, and trace logs.

3. Architecture / Algorithms: Each operator role is backed by one LLM model, and the whole team shares the same model in each run. Operators communicate actions, reports, queries, and approval requests via a typed message bus. Actions have deterministically derived authority levels, and high-impact actions require explicit approval via a mock human approval console, simulating two-person integrity checks. Guardrail layers implement optional policy enforcement rules on actions and messages, including sender-role gating, veto channels, procedure citation enforcement, and rate limiting. The Safety Technical Advisor agent plays an independent advisory or veto role in different experiment conditions.

4. Training Regime: Not applicable as operators use pretrained LLMs from vendors or open-source models without further fine-tuning here. The attack workload is created by a separate instruction-tuned LLM agent conditioned on previous turn summaries and fixed strategies.

5. Evaluation Protocol: Candidate operator models are evaluated by replaying all fixed attack sessions under defined scenario initial states (normal operation or accident conditions), guardrail and advisor modes, and attacker visibility assumptions. The primary metric is ASRCSF — fraction of sessions causing any CSF loss — an objective plant-state derived safety breach rather than text classification. Additional turn-level diagnostic scores from an auxiliary fixed LLM judge are used for breach localization but not summary scoring. Paired replay of all operators against the same attack logs enables fine-grained comparative ranking and failure-set overlap analysis. Extensive ablations isolate effects by scenario, guardrail stack, advisor authority level, and attacker visibility.

6. Reproducibility: The authors provide open source the simulation environment, full attack dataset, and replay tooling, enabling independent evaluation of candidate LLM operator agents against a physically grounded, multi-agent, multi-turn red-teaming benchmark. Attack generation internals and LLM models backing attackers are proprietary, but replay suffices for comparative studies. All trace logs are cryptographically hashed for tamper-evidence.

Example End-to-End: In one session, an attacker uses the CH-B channel to impersonate internal management escalating a request to disable safety alarms gradually over multiple turns, leveraging authority spoofing and urgency injection strategies. The LLM operator team exchanges messages proposing and approving actions, mediated by guardrails and human approval mock console. After several turns, one CSF (e.g., heat sink integrity) transitions from intact to lost, terminating the run and attributing the loss to the corresponding attacker payload. This sequence is logged exhaustively and can be replayed against different operator models to see which resist or fail similarly.

Technical innovations

• A multi-turn, multi-agent red-teaming benchmark grounded in a safety-critical physical simulation with objective harm signals rather than text-based heuristic or LLM-juried definitions.
• Use of a configurable five-role operator team architecture enforcing role-specialized authority hierarchies, two-person integrity mock approval, and layered guardrail policy stacks.
• Introduction of four distinct adversarial ingress channels modeling different realistic compromise modes with bounded adaptive messaging.
• A fixed-attack replay protocol enabling deterministic, comparable evaluation of any candidate operator stack against a shared adversarial workload with per-turn harm attribution.

Datasets

• NRT-Bench attack workload — 149 multi-turn attack sessions — proprietary/internal generation, released with replay tooling

Baselines vs proposed

• gpt-5.4-mini (no guardrails, advisory on, normal scenario): ASRCSF = 18.1% vs full guardrail, advisory on: 1.4%
• claude-haiku-4-5 (no guardrails, advisory on, normal scenario): ASRCSF = 33.3% vs full guardrail, advisory on: 43.1%
• gpt-5.4-mini vs claude-haiku-4-5 aggregate ASRCSF across all cells: 23.7% vs 37.5%
• gemma-4-26B vs Qwen3.5-35B on paired replay test split ASRCSF: 12.1% vs 11.4%
• Authority spoofing attack family ASRCSF: gpt-5.4-mini 37.8% vs claude-haiku-4-5 54.2%
• Gradual escalation attack family ASRCSF: gpt-5.4-mini 29.2% vs claude-haiku-4-5 50.3%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.20408.

Fig 1: NRT-Bench at a glance. A multi-turn red-team attacker probes a safety-critical control-room venue through four ingress

Fig 2: Attack-agent I/O contract. The agent consumes a goal, a strategy-family label, the session history, and the previous

Fig 3 (page 2).

Limitations

• Attack workload is fixed post-hoc and generated against a seed defender, which may bias evaluation and underestimate worst-case adaptive vulnerability of novel models.
• Only black-box attacker visibility considered as default; white-box visibility experiments not fully reported for all operators.
• The simulated plant and operator roles are abstractions without physical hardware-in-the-loop or real control systems validation; results depend on simulation fidelity.
• Operators are single-model teams rather than heterogeneous multi-model teams, limiting insights into diversity or ensemble safety strategies.
• Guardrail and approval abstractions do not model full human-in-the-loop complexity and may not capture real-world operator error modes or adversarial social engineering fully.
• No adversarial training or defensive fine-tuning of operator models evaluated; purely zero-shot robustness testing.

Open questions / follow-ons

• How would operator-team heterogeneity or ensemble approaches affect robustness and failure overlap compared to single-model teams?
• Can adaptive adversarial training or online attack mitigation via real-time defense layers significantly reduce ASRCSF without impairing operator functionality?
• How do attacker visibility and information leakage affect attack success rates in more realistic threat models beyond black-box and white-box extremes?
• What is the impact of richer human-in-the-loop modeling, including real operator distraction, fatigue, and social engineering, on system vulnerability?

Why it matters for bot defense

This work provides a principled evaluation methodology for assessing large language model agents deployed in safety-critical, multi-agent supervisory roles beyond simple text-based content moderation or single-turn jailbreak detection. For bot-defense and CAPTCHA practitioners, NRT-Bench illustrates the challenges in robustly defending multi-agent LLM systems against sustained, adaptive adversarial inputs aiming to induce unsafe behaviors. The finding that defense stacks and safety advisor layers can help one model while harming another emphasizes the difficulty of one-size-fits-all automated guardrails and the need for role-aware, model-specific adaptation. The benchmark's objective harm signal, based on physical-system simulation rather than heuristic classifiers, sets a new standard for evaluating real-world impact of adversarial attacks on AI-driven operations. Practitioners designing bot defenses for interactive LLM agents should consider multi-turn, multi-agent threat models and evaluate defenses with agent-level compositional context rather than isolated prompt classification alone.

Cite

@article{arxiv2606_20408,
  title={ LLM agent safety, multi-turn red-teaming, jailbreak benchmarks, adversarial robustness, safety-critical systems },
  author={ Hanwool Lee and Dasol Choi and Bokyeong Kim and Seung Geun Kim and Haon Park },
  journal={arXiv preprint arXiv:2606.20408},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.20408}
}

Read the full paper

• arXiv:2606.20408 (abstract)
• arXiv:2606.20408 (PDF)