Pretraining Data Can Be Poisoned through Computational Propaganda

Source: arXiv:2607.15267 · Published 2026-07-16 · By Victoria Graf, Hannaneh Hajishirzi, Noah A. Smith, David Kohlbrenner, Kyle Lo

TL;DR

This paper investigates the feasibility and impact of poisoning language model (LM) pretraining data through large-scale third-party content injection on the open web. Unlike prior poisoning studies that focus on small, well-curated sources such as Wikipedia, the authors analyze heterogeneous web-scale corpora typically sourced from web crawls like Common Crawl. They identify public discussion interfaces—particularly comment sections of webpages—as a practical and large attack surface where adversaries can inject malicious or misleading content. To quantitatively assess whether poisoned content survives the complex data curation pipeline (including crawling, extraction, deduplication, language and quality filtering), the paper introduces HALFLIFE, a novel probabilistic framework to estimate poison inclusion in final LM training corpora.

Using HALFLIFE, the authors find that injected comments survive crawling and filtering with a non-negligible probability (~0.13%), which is enough to impact more documents than the entire Wikipedia subset in certain representative datasets. Controlled experiments with Olmo-3-like pretraining show that trace token poison rates of 0.001-0.1% suffice to induce measurable model biases favoring attacker-chosen narratives even after instruction tuning, although the effect diminishes with model scale and fine-tuning. The study also rules out programmatic ads as an ineffective poison vector due to their removal by scraping pipelines. Beyond empirical findings, the paper discusses defense techniques such as comment-aware extraction and provenance-aware filtering to mitigate poisoning risks pervasive in web-scale, heterogeneous training data.

Overall, the paper reveals an important and previously underexplored avenue for injecting adversarial content into LM pretraining data via publicly accessible web infrastructure, supported by a rigorous quantitative analysis framework and realistic threat model. It provides new insights into how model training pipelines may unwittingly consume and amplify third-party propagandistic content, underscoring the need for enhanced poison-aware data curation to ensure model reliability and safety.

Key findings

• Public discussion comments appear on 3.4% of webpages in sampled Common Crawl data (CC-MAIN-2025-51 shard, 181,857 pages).
• Approximately 71.9% of injected comment content survives crawling and text extraction (P(captured|injectable) = 0.719).
• Over 5.5% of injected poisoned comments survive through heuristic, language, and quality filtering steps (P(not filtered|captured, injectable) = 0.055).
• End-to-end estimated poison inclusion probability (post crawling and filtering) is about 0.13% of documents, exceeding the ~0.067% fraction that Wikipedia contributes to Dolma 3 data.
• Experiments show poisoning with 0.001-0.1% token poison rates in pretraining shifts model completion preferences toward attacker-favored entities by 3-20 percentage points over clean baselines, varying by model size (65M to 1.3B parameters).
• Instruction fine-tuning reduces the retention of poison effects, especially for larger models where SFT retention drops from ~40% at 65M to under 15% at 709M and 1.3B parameters.
• More stealthy poison injection formats without explicit chat markers (Q/A and no-label) still significantly contaminate base models, though their effects diminish more after instruction tuning compared to explicit USER/ASSISTANT format.
• Programmatic advertisements are a poor vector for poisoning as injected content rarely survives crawling and text extraction due to their web infrastructure characteristics.

Threat model

The adversary is a web-level actor with capabilities limited to injecting content onto third-party webpages via public discussion interfaces (comments) without ownership or administrative control. They do not have insider access to pretraining data collection or model training infrastructure. The adversary cannot select which pages are crawled or directly influence data filtering processes, rendering their data injection influence probabilistic and indirect. Their objectives can be to manipulate future LM outputs toward attacker-controlled narratives or degrade model quality.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is assumed to have minimal knowledge and no direct control over pretraining datasets or pipelines. They act as web users or content developers with the capability to inject content via public web interfaces, such as automated commenting systems leveraging browser automation tools (e.g., Selenium). They cannot predict which web pages are crawled or directly influence data curation pipelines. The adversary’s goals include altering model outputs toward specific narratives or degrading model quality (e.g., inducing harmful outputs).

2. Data & Experimental Setup: The authors analyze sampled webpages from 200 WARC files in Common Crawl’s CC-MAIN-2025-51 shard, totaling 181,857 web pages, using signatures to identify pages with comment platforms.

For poisoning experiments, they simulate poisoning by injecting crafted poison text (question/answer pairs averaging 37.5 words) replacing real comments in sampled pages to avoid live injections.

They use text extraction tools (Resiliparse) typical of LM data pipelines, filtering heuristics, and quality classifiers from Dolma 3 pipeline for survival estimation.

Model poisoning experiments fine-tune Olmo-3-like models at diverse scales (65M to 1.3B parameters). Poison content targets three entity pairs with multiple prompt-completion pairs (60 each), injected at token poison rates 0.1%, 0.01%, and 0.001% alongside Dolma 3 web data (≈2×Chinchilla tokens). Instruction fine-tuning is applied post-pretraining.

3. Architecture & Algorithms: No novel LM architecture introduced; models follow Olmo-3 specifications matching Dolma 3 settings. Novelty lies in the HALFLIFE probabilistic framework estimating poison inclusion as the product of the probability of page injectability (web openness for comment injection), likelihood injected content is captured by crawling and text extraction, and probability content survives filtration.

4. Training Regime: Models pretrained from scratch mixing poison data at trace rates with clean Dolma 3 web corpus. Instruction fine-tuned using supervised dialogues subsets. Training hyperparameters include learning rates (LR=8e-5 for SFT), batch sizes (524k tokens for SFT), and training epochs (2 epochs supervised fine-tuning). Random seed details not specified explicitly.

5. Evaluation & Metrics: HALFLIFE computes end-to-end inclusion probabilities: P(include) = P(injectable) × P(captured|injectable) × P(not filtered|captured, injectable).

Effectiveness of poison evaluated by model likelihood preference for attacker-favored completions using log-likelihood scoring over held-out test pairs (20 pairs per entity set), reporting % preference values.

Comparative ablations across model scale, poison rate, and poison injection formats (USER/ASSISTANT dialogue style vs Q/A vs no-label).

6. Reproducibility: Code for HALFLIFE analysis is open-sourced at https://github.com/VictoriaGraf/HalfLife. Pretraining datasets rely on Dolma 3 web subsets, publicly referenced. Exact model weights or training scripts for poisoned models are not explicitly released.

7. Example End-to-End: An attacker finds injectable webpages (3.4% of sampled Common Crawl pages have public comment interfaces). They inject poison comments (question/answer pairs) simulating targeted content. Crawlers extract ~72% of these injected comments. Filtering pipelines, heuristics, and quality classifiers remove many but allow ~5.5% of injected poison to persist into final pretraining data, resulting in 0.13% overall inclusion.

Models pretrained on this data manifest statistically significant biases towards attacker narratives, confirming the practical viability of the entire injection-to-model-impact chain.

Technical innovations

• HALFLIFE: a novel probabilistic framework to estimate adversarial poison inclusion probability across injection, crawling, extraction, and curation stages.
• Identification of third-party content injection via public discussion interfaces as a scalable vector for pretraining data poisoning beyond small curated sources like Wikipedia.
• Empirical demonstration that injected comment content survives typical large-scale LM data extraction and filtering pipelines at non-trivial rates.
• Controlled scaling experiments showing trace token poison rates in pretraining data induce measurable, format-robust biases in both base and instruction-tuned models.

Datasets

• Common Crawl CC-MAIN-2025-51 — ~181,857 webpages sampled for comment platform analysis — publicly available
• Dolma 3 web subset — large-scale pretraining corpus from web data — publicly referenced
• Synthetic poisoning datasets — 60 prompt-completion pairs for each of 3 entity pairs (Citroen vs Renault, Boeing vs Airbus, Pfizer vs Moderna) — constructed by authors

Baselines vs proposed

• Clean baseline model preference for attacker entity ranges from 51.3% to 56.5% across sizes 65M to 1.3B.
• At 0.1% token poison rate, base models show up to +20.1% preference increase over baseline (e.g., 150M model from 53.2% to 73.3%).
• Instruction fine-tuning reduces poison effect retention, dropping from ~40% retention at 65M scale to under 15% at 709M and 1.3B scales (Table 1).
• More stealthy poison formats (Q/A, no-label) produce comparable contamination in base models but reduced effect retention post instruction-tuning relative to USER/ASSISTANT format (Table 2).
• Programmatic advertisement injection shows near-zero poison inclusion due to scraping pipeline incompatibility, unlike comments.

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.15267.

Fig 1: Diagram of the attack vector as it passes through the pretraining data pipeline. Poisoned

Fig 2: Number of webpages in which an attacker must attempt to inject poison (y) as a function

Fig 7: Side-by-side comparison of the same page rendered without JavaScript (left) and with

Limitations

• Use of Common Crawl as proxy may not fully capture proprietary crawler behaviors or data curation pipelines at commercial AI labs.
• Comment injection analysis relies on detection in static HTML and sandboxed injections rather than live attacks on real websites—may overestimate injection feasibility.
• Analysis is limited to a single data filtering pipeline (Dolma 3) and its specific heuristics and quality classifiers; other pipelines may have different survival rates.
• No adversarial evaluation against adaptive defenses or dynamic crawling/filtering strategies that could diminish poison survival.
• No formal robustness or differential testing of poisoning impact across diverse model architectures or fine-tuning regimes beyond Olmo-3-like models.

Open questions / follow-ons

• How can more targeted or adaptive poisoning strategies exploit knowledge of specific data pipelines or crawling policies to increase poison inclusion?
• Can comment-aware or provenance-aware filtering methods be effectively integrated into large-scale LM data pipelines without compromising dataset quality or scale?
• What are the poisoning risks and mitigation strategies for low-resource or non-English language models where web data diversity and volume differ?
• How resilient are advanced model architectures or alternative training regimes (e.g., RLHF, instruction tuning) to stealthy poisoning attacks delivered via web comments?

Why it matters for bot defense

From a bot-defense and CAPTCHA engineering perspective, this paper highlights a novel adversarial vector exploiting publicly accessible web interfaces—comment sections—to inject poisoned data at scale into LM pretraining corpora. This underlines a broader security challenge for platforms hosting public content: automated bots can be weaponized not just to affect human users, but indirectly to degrade or manipulate AI models trained on web data. Understanding this attack chain emphasizes the importance of integrating bot detection, rate limiting, and authentication controls on open comment or discussion interfaces to disrupt large-scale poisoning attempts at the source.

Additionally, CAPTCHA practitioners should be aware that mere openness in platform interaction mechanisms contributes to poisoning risk. Beyond direct user-facing security, maintaining the integrity of AI training data demands cooperation between web platform operators and dataset curators. Mitigation strategies such as comment-aware extraction or provenance tracking highlighted by the paper could inform future CAPTCHA deployment policies and platform-level anti-abuse designs tailored to break the link between automated content injection and LM data ingestion.

Cite

@article{arxiv2607_15267,
  title={ Pretraining Data Can Be Poisoned through Computational Propaganda },
  author={ Victoria Graf and Hannaneh Hajishirzi and Noah A. Smith and David Kohlbrenner and Kyle Lo },
  journal={arXiv preprint arXiv:2607.15267},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.15267}
}

Read the full paper

• arXiv:2607.15267 (abstract)
• arXiv:2607.15267 (PDF)