DataShield: Uncovering Risky Fine-Tuning Data Across LLMs Through Consensus Subspace Alignment

Source: arXiv:2607.15081 · Published 2026-07-16 · By Zefeng Wu, Weiwei Qi, Jielong Chen, Tianhang Zheng, Di Hong, Chaochao Lu et al.

TL;DR

This paper addresses the problem that fine-tuning large language models (LLMs) on domain-specific data, even innocuous data, can degrade their safety behavior, causing increased compliance with harmful or unsafe queries. Prior methods to identify risky fine-tuning samples rely on single-model mean vector representations of safety directions, limiting risk assessment effectiveness and transferability across models. DataShield proposes a novel multi-model consensus subspace alignment approach that builds joint safe and unsafe semantic subspaces from multiple safety-aligned LLMs with diverse architectures and tokenizers. It estimates sample-level and tokenizer-agnostic segment-level risk by measuring relative alignment with these consensus subspaces. Extensive experiments across multiple target models show DataShield reduces harmful attack success rates (ASR) by 14.6% with sample filtering and 32.3% with segment masking versus prior state-of-the-art baselines, while preserving downstream task utility. The approach also avoids target-model-specific risk computations, improving reuse and reducing preprocessing cost.

Key findings

• DataShield sample filtering reduces average ASR by 14.6% compared to SOTA baselines across four target LLMs fine-tuned on Alpaca and Dolly datasets.
• DataShield segment masking reduces average ASR by 32.3% compared to token-level masking baseline TOSS while maintaining similar downstream utility (SLIMORCA metric).
• DataShield achieves lower ASR than baselines that use target-model-specific risk signals, e.g. on Gemma2-27B-it, DataShield-Sp reduces HARM ASR from 28.0% (SOT) to 18.6%.
• Autoregressive risk decoupling localizes risky segments more accurately, reducing ASR from 14.1% to 9.4% on HarmBench and from 11.9% to 7.7% on HEx-PHI benchmarks.
• Using consensus subspaces built from 3 source models (Llama3-8B, Qwen2.5-7B, Mistral-7B) is sufficient to gain strong transfer performance; adding more models yields diminishing returns.
• The proposed unsafe-versus-safe subspace alignment gap outperforms one-sided or mean vector gap risk scoring, e.g. reducing PHI ASR from 19.3% to 7.6% on Qwen3-4B-it.
• DataShield preprocessing requires less GPU memory (47.9–58.1 GB) and time (64–118 minutes) than competing baselines when processing ~14K examples.
• The tokenizer-agnostic segment-level masking enables risk localization transferable across models with different tokenizations, unlike prior token-level methods.

Threat model

The adversary introduces or influences fine-tuning data that may degrade the safety capabilities of aligned LLMs by increasing compliance with harmful queries. The defender’s system cannot retroactively modify trained source models or rely on target-model-specific information during risk estimation, and adversaries cannot manipulate the consensus source models or tokenization used for evaluation. The focus is on preemptive identification and removal or masking of risky fine-tuning data prior to fine-tuning on diverse downstream target LLMs.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is an implicit participant via unsafe fine-tuning data that degrades LLM safety. The adversary cannot manipulate trained source models or tokenizers. The defender seeks to identify risky fine-tuning data before downstream fine-tuning to preserve safety across diverse target models with varying architectures and tokenizers. Target models are unseen during risk estimation.

2. Data: The main fine-tuning task datasets are Alpaca and Dolly instruction-tuning datasets. Safety evaluation uses harmful behavior benchmarks HEx-PHI and HarmBench. Utility is measured on a subset of SLIMORCA. Probing data with paired harmful and safe behavior examples is used to build safe/unsafe subspaces.

3. Architecture/Algorithm: DataShield collects 'safety-critical' representations from the top N=3 critical layers of multiple (K=3) safety-aligned source models (Llama3-8B-Instruct, Qwen2.5-7B-Instruct, Mistral-7B-Instruct). For each model, it extracts compact hidden states using layer representation combination for each example. It builds safe and unsafe covariance operators from hidden states of safe and unsafe probing examples and performs spectral decomposition to form orthonormal safe and unsafe subspace bases.

Risk scores per sample or per semantic segment are computed by measuring the normalized squared projection alignment onto these consensus safe and unsafe subspaces within each model's space. The sample risk is the average difference in alignment to unsafe minus safe subspace across models. Segment-level risk is similarly computed by splitting responses into tokenizer-independent raw text segments and mapping them to token indices per model. To prevent risk smearing caused by autoregressive context, an incremental alignment delta is computed per token and the maximum delta in each segment is used. This produces tokenizer-agnostic, fine-grained risk localization.

4. Training Regime: Target models are fine-tuned with LoRA on filtered or masked datasets under a global intervention budget ρ=0.2 (top 20% risky samples or segments removed/masked). Multiple runs with different seeds are averaged. No training of DataShield itself is required beyond subspace construction.

5. Evaluation Protocol: Attack Success Rate (ASR) on harmful benchmarks HEx-PHI and HarmBench is the main safety metric, scored by GPT-4o with Gemini-3.1-Pro validation. Utility is measured on SLIMORCA instruction-following. Multiple baseline filtering/masking methods (Random, Bi-Anchor, SEAL, LARF, SOT, TOSS) are compared on Alpaca and Dolly data across four unseen target LLMs, measuring transferability. Ablations evaluate effects of consensus model number, risk decoupling, subspace dimensionality, and subspace scoring variants.

6. Reproducibility: The source code is publicly released at https://github.com/ZJU-LLM-Safety/DataShield. Target model weights and datasets (Alpaca, Dolly) are publicly available. Experimental details including segment construction, layer selection, and hyperparameters are provided in the appendix. However, some proprietary target models are closed.

Example: For a given fine-tuning sample, DataShield extracts representations from critical layers of each source model, projects onto learned safe and unsafe subspaces, computes alignment gap scores per model, averages across K=3 models to get a consensus sample risk score. Samples exceeding risk thresholds are removed; for segment masking, response text is split into tokenizer-independent spans, risk increments per token tracked and aggregated to produce fine-grained masks applied during target fine-tuning loss.

Overall, the method leverages multi-model consensus latent subspaces and tokenizer-independent segmentation to achieve transferable and interpretable fine-grained risk estimation for safety-preserving filtering and masking of fine-tuning data.

Technical innovations

• Construction of consensus safe and unsafe semantic subspaces via spectral decomposition from multiple safety-aligned LLMs instead of single-model mean vectors.
• Tokenizer-agnostic, segment-level fine-grained risk localization through raw text segmentation and incremental alignment score decoupling to remove autoregressive context carry-over effects.
• Use of alignment gap between safe and unsafe subspaces as a risk measure, which outperforms prior one-sided or mean-gap approaches.
• Multi-model consensus scoring enables risk estimates transferable across diverse target LLM architectures and tokenizers without requiring target-model-specific risk computations.

Datasets

• Alpaca — ~52K samples — public instruction dataset
• Dolly — ~15K samples — public instruction dataset
• HEx-PHI — Harmful query benchmark for evaluation — public
• HarmBench — Harmful behavior benchmark for evaluation — public
• SLIMORCA — Instruction-following utility benchmark subset — public

Baselines vs proposed

• Standard fine-tuning: ASR HARM ~76.0% (Phi3-medium) vs DataShield-Sp 17.1% and DataShield-Sm 17.1%
• Bi-Anchor: ASR HARM ~45.2% (Alpaca, multiple models) vs DataShield-Sp ~23.5%
• SEAL: ASR HARM ~42.5% (Alpaca, multiple models) vs DataShield-Sp ~23.5%
• LARF: ASR HARM ~34.1% (Alpaca, multiple models) vs DataShield-Sp ~23.5%
• SOT: ASR HARM ~23.7% average vs DataShield-Sp ~15.1%
• TOSS token masking: ASR HARM ~40.8% vs DataShield-Sm ~23.1%
• DataShield-Sp preprocessing memory 47.9 GB, time 64 min vs Bi-Anchor 168 GB, 503 min
• DataShield-Sm preprocessing memory 58.1 GB, time 118 min vs TOSS 242 GB, 675 min

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.15081.

Fig 1: Overview of DataShield. DataShield constructs safety-critical semantic spaces from multiple safety-

Fig 2 (page 3).

Limitations

• Evaluations are limited to supervised fine-tuning on instruct datasets; other adaptation methods like RLHF are not considered.
• Experiments cover only a few model families and languages; applicability to multilingual or other-domain-specific models remains to be studied.
• Safety evaluation focuses on known harmful benchmarks; behavior in specialized domains (legal, medical, financial) may differ.
• Segment-level masking granularity depends on sentence and punctuation heuristics which may have limitations for complex text spans.
• Marginal gains plateau after 3 source models, raising questions on scalability to very large model pools.
• No adversarial noisy data or active attacks evaluated; robustness against adversarial fine-tuning remains unexplored.

Open questions / follow-ons

• How does DataShield perform on multilingual data and non-English language models for safety preservation?
• Can the consensus subspace approach be integrated with reinforcement learning fine-tuning (RLHF) pipelines?
• What is the robustness of DataShield against adaptive adversaries attempting to evade risk detection by crafting fine-tuning data?
• How can the segmentation and risk scoring method be refined for domain-specific safety risks (e.g., medical misinformation) where harmfulness is more subtle?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, DataShield presents a rigorous method to preemptively analyze and filter fine-tuning datasets that could degrade safety controls of deployed language models. Its multi-model consensus methodology ensures that risk assessments do not overfit to a single model architecture or tokenizer, increasing their robustness and transferability when adapting models for sensitive application domains. The segment-level tokenizer-agnostic masking offers a way to suppress unsafe knowledge granules without wholesale example removal, preserving utility while strengthening alignment.

While not directly a CAPTCHA technology, the principles of consensus-based latent subspace alignment for risky content detection and fine-grained, tokenizer-independent masking could inspire defense mechanisms against adversarial query generation or content poisoning attacks in interactive language understanding systems. The work highlights the importance of cross-model safety signals and granular risk localization, useful considerations when designing CAPTCHA or bot-defense schemes reliant on language models.

Cite

@article{arxiv2607_15081,
  title={ DataShield: Uncovering Risky Fine-Tuning Data Across LLMs Through Consensus Subspace Alignment },
  author={ Zefeng Wu and Weiwei Qi and Jielong Chen and Tianhang Zheng and Di Hong and Chaochao Lu and Liang He and Zhan Qin and Kui Ren },
  journal={arXiv preprint arXiv:2607.15081},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.15081}
}

Read the full paper

• arXiv:2607.15081 (abstract)
• arXiv:2607.15081 (PDF)