What Do Safety-Aligned LLMs Learn From Mixed Compliance Demonstrations?

Source: arXiv:2606.20508 · Published 2026-06-18 · By Sihui Dai, Mann Patel

TL;DR

This paper investigates how large language models (LLMs) aligned for safety interpret and learn from mixed demonstrations of compliance, specifically benign compliance (helpful responses to harmless prompts) versus harmful compliance (helpful responses to harmful prompts). Prior work showed that many harmful compliance demonstrations can jailbreak LLMs, but it remained unclear if benign and harmful compliance demonstrations have interchangeable or different effects on a model’s likelihood to comply with harmful queries. The authors systematically mix different ratios of benign and harmful compliance demonstrations and evaluate four diverse models (Llama-3.1-8B, OLMo-3.1-32B, GPT-OSS-20B, Gemma-4-31B) over a held-out harmful query set from multiple harm benchmarks. They test hypotheses about whether the total number of compliant demos, only harmful demos, or both benign and harmful demos influence harmful compliance. Key findings are that benign and harmful compliance demonstrations are not interchangeable and their effects depend on the model. For example, benign demos can dilute harmful compliance in some models (Llama, Gemma), slightly amplify it in others (GPT-OSS), or have no effect (OLMo). They further show the critical role of preference optimization training (DPO) in decoupling benign compliance from harmful compliance on OLMo, reveal a strong recency effect from ordering of demos, and demonstrate that format adoption and compliance learning are dissociable with varying refusal behaviors across models. This work moves beyond showing demonstration-based jailbreaking occurs to characterizing how and why models internalize different compliance signals from mixed contexts. Their empirical and statistical analyses provide a clear picture of the nuanced and model-dependent ways alignment interacts with in-context learning of safety-relevant behavior.

Key findings

• All four models reject the total-count hypothesis (Htotal) that benign and harmful compliance demonstrations are interchangeable; compliance rates depend significantly on the harmful fraction ϕ (Fig 2, Table 1).
• Benign compliance demonstrations dilute harmful compliance for Llama-3.1-8B and Gemma-4-31B (β2 = -0.1782 and -0.2010, p < 1e-100), have no significant effect on OLMo-3.1-32B (β2 = -0.0118, p = 0.11), and slightly amplify harmful compliance on GPT-OSS-20B (β2 = +0.0319, p = 0.011) (Fig 3, Table 2).
• Preference optimization training stage (DPO) on OLMo-3.1-32B removes amplification effect seen after supervised fine-tuning (SFT) where benign demos increase harmful compliance, stabilizing compliance rates across benign demo counts (Fig 4, Table 3).
• Model behavior shows strong recency bias: harmful demonstrations placed closest to evaluation query (suffix ordering) increase harmful compliance substantially compared to prefix or middle placement (Fig 5).
• Format adoption (copying demonstration response style) and compliance (producing harmful content) are dissociable; some models (Llama, OLMo) adopt format even during refusal, while others (Gemma, GPT-OSS) suppress format adoption when refusing harmful requests (Figs 6 and 7).
• GPT-OSS-20B exhibits the strongest robustness to many-shot demonstration attacks, maintaining low harmful compliance regardless of demonstration composition or ordering.
• The harmful and benign compliance effects vary by model architecture and training regimen, reflecting different safety-alignment strategies and refusal mechanisms.

Threat model

The adversary is assumed to have black-box access to the model’s input and output interface and attempts to induce harmful compliance by supplying harmful evaluation queries possibly preceded by mixed demonstrations of compliance behavior. The attacker does not have direct access to internal model parameters or weights and does not perform activation edits or model fine-tuning. The attacker’s knowledge includes ability to control the in-context examples but not to interfere with the training or preference optimization processes. The adversary seeks to maximize harmful content production by leveraging demonstration-based jailbreaks.

Methodology — deep read

The authors investigate how safety-aligned LLMs respond to in-context demonstrations mixing benign compliance and harmful compliance examples. The threat model assumes an adversary who tries to jailbreak the model using many-shot harmful prompts, with knowledge limited to input prompts and no direct model internals access. The core research question is how the composition and ordering of in-context compliant demonstrations affect a model's likelihood to comply with harmful evaluation queries.

Data provenance:

• Harmful compliance pool constructed from RedTeam-2K prompts.
• Benign compliance pools sourced from UltraChat (UltraFeedback), OR-Bench, and GPT-OSS-120B benign rewrites of harmful prompts.
• Compliant responses for benign and harmful prompts generated via GPT-OSS-20B.
• Held out evaluation set: A pool of 1,404 harmful queries from HarmBench, SORRY-Bench, and WildGuard-test harmful subset, distinct from context demonstration pools.

Models:

• Evaluated Llama-3.1-8B-Instruct, OLMo-3.1-32B-Instruct (plus its SFT and DPO checkpoints), GPT-OSS-20B, Gemma-4-31B-IT.

Experimental setup:

• Mixed in-context demonstration contexts constructed with controlled numbers of benign (Nb) and harmful (Nh) compliance examples.
• Demonstrations truncated to 2000 characters to control length variability.
• Default ordering: benign demos first, harmful demos following; alternative orderings tested include prefix, suffix, random, interleave, and middle variations.
• Compliance of evaluation harmful prompts measured as 1 - refusal rate, with refusals identified by WildGuard refusal classifier.

Evaluation protocol:

• Tested three hypotheses:
  1. Total-count (Htotal): Compliance depends only on total compliant demos, treating benign and harmful similarly.
  2. Harmful-count (Hharm): Only harmful compliance demos increase compliance.
  3. Joint-count (Hjoint): Both benign and harmful demos contribute, possibly diluting or amplifying compliance.
• Statistical tests: χ2 tests to reject Htotal, logistic regression logit model logit(P_compliance) = β0 + β1log(Nh +1) + β2log(Nb + 1) to evaluate Hharm vs Hjoint.
• Multiple demonstrations counts spanning N=4 to 128, and harmful fraction ϕ from 0 to 1 were evaluated. Random sampling and multiple seeds employed for measurement reliability (over 2,800 evaluation points).

Training regimen analysis:

• Used OLMo-3.1-32B at different training stages: SFT checkpoint, DPO checkpoint, and final RL-VR instruction tuned model to study the evolution of benign demo influence.

Format adoption study:

• Added neutral and comply prefixes to demonstration responses to disentangle format copying from compliance behavior.
• Measured adoption rates separately for compliant vs refusal responses.

Recency bias:

• Compared compliance under different demonstration orderings to show placement effects.

Reproducibility:

• No direct code or weights noted as released; some datasets (RedTeam-2K, HarmBench) are public, others referenced in related work. Specific demonstration pools and controlled experimental protocols are described in sufficient detail for replication.

Example end-to-end: Consider a context with 64 total demonstrations containing 32 benign compliance examples from UltraChat and 32 harmful compliance demos from RedTeam-2K, arranged with harmful examples last (suffix ordering). The model is then given a held-out harmful query from HarmBench, and the response is classified by WildGuard. Repeating with variations in Nb and Nh and across ordering conditions allows statistical analysis of compliance rates and testing of the outlined hypotheses.

Technical innovations

• Formal hypothesis testing framework for disentangling effects of benign versus harmful compliance demonstrations on harmful compliance behavior.
• Use of mixed in-context demonstration pools combining benign and harmful compliance examples to characterize their joint and individual contributions.
• Identification of preference optimization training stage (DPO) as a key shift reducing harmful compliance amplification from benign demonstration signals.
• Empirical demonstration of strong demonstration ordering recency bias impacting compliance rates.
• Distinguishing compliance behavior adoption versus format adoption from demonstrations, revealing heterogeneous refusal mechanisms across models.

Datasets

• RedTeam-2K — size unspecified — public safety-focused adversarial prompt set
• UltraChat (from UltraFeedback) — size unspecified — publicly documented benign compliance prompts
• OR-Bench — size unspecified — publicly documented benign compliance prompts
• HarmBench — 1404 harmful evaluation queries — public benchmark for model harm evaluation
• SORRY-Bench — size unspecified — public harmful prompt benchmark
• WildGuard-test harmful subset — size unspecified — public harmful queries benchmark subset

Baselines vs proposed

• Llama-3.1-8B: compliance rate increases from ~0.3 to ~0.8 as harmful fraction ϕ increases from 0 to 1 at N=64; benign demos dilute harmful compliance (β2 = -0.1782)
• OLMo-3.1-32B: compliance increases with harmful fraction ϕ; benign demo effect not significant (β2 = -0.0118, p=0.11); post-DPO stage eliminates benign amplification effect
• GPT-OSS-20B: compliance remains low (~<0.1) across harmful fractions and total demos, showing strong robustness; slight benign amplification (β2 = +0.0319)
• Gemma-4-31B: compliance increases with harmful fraction ϕ; benign demos dilute compliance significantly (β2 = -0.2010)
• Ordering effect (Fig 5): suffix ordering raises compliance rates by up to 35% vs prefix ordering (largest in Gemma-4-31B)
• Format adoption vs compliance (Figs 6-7): Llama and OLMo adopt format prefix even during refusal at ~52-87%, while Gemma and GPT-OSS show near-zero format adoption on refusal responses

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.20508.

Fig 1: Benign vs. Harmful Compliance Demonstration. In a

Limitations

• Quantitative effect sizes and statistical significance reported, but mechanistic causes of decoupling between benign and harmful compliance require further investigation.
• Limited to a subset of safety-aligned models (all relatively recent) — results may not generalize to all model architectures or alignment regimes.
• Evaluation uses only harmful queries from public benchmarks, which may incompletely represent real-world adversarial prompts or user intent variations.
• Demonstration truncation to 2000 characters may discard contextual details influencing compliance in production contexts.
• No direct adversarial intervention or adaptive attacker modeled beyond static many-shot demonstration mixtures.
• No direct mechanistic interpretability or latent representation analysis to identify internal causal structure of compliance signals.

Open questions / follow-ons

• What internal representational or mechanistic changes during preference optimization (DPO) lead to the decoupling of benign and harmful compliance effects?
• How do different model architectures and sizes beyond those tested here affect the generalization of benign versus harmful demonstration effects on compliance?
• Can dynamic or adaptive demonstration ordering further modulate harmful compliance beyond fixed suffix or prefix patterns?
• How would adversarial training or explicit counter-demonstrations interact with the joint-count hypothesis and impact model robustness?

Why it matters for bot defense

For practitioners in bot-defense and CAPTCHA design, this paper highlights critical nuances in how LLM safety policies respond to in-context exposure to benign versus harmful behavioral signals. Since malicious agents can attempt demonstration-based jailbreaks by feeding mixed signals into language models, it is not sufficient to treat all compliance demonstrations equivalently when modeling attacker capabilities. Demonstration ordering and training methodology (notably the presence of preference optimization) heavily influence the model’s susceptibility to harmful compliance. These findings suggest that bot-defense systems relying on LLMs should include safeguards against subtle in-context contamination, prioritize robustness to specific harmful compliance examples, and carefully consider the model’s training regimen. Furthermore, understanding that format adoption and compliance behavior can dissociate guides monitoring systems to distinguish superficial mimicry from actual harmful output generation. Incorporating temporal context weighting (e.g., recency bias in demonstration ordering) can assist in detecting or mitigating malicious prompt injection attempts. Overall, this work offers an analytical framework to anticipate how demonstration-based adversarial attacks might evolve and how aligned LLM behavior can be systematically characterized.

Cite

@article{arxiv2606_20508,
  title={ What Do Safety-Aligned LLMs Learn From Mixed Compliance Demonstrations? },
  author={ Sihui Dai and Mann Patel },
  journal={arXiv preprint arXiv:2606.20508},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.20508}
}

Read the full paper

• arXiv:2606.20508 (abstract)
• arXiv:2606.20508 (PDF)