Out of Sight: Compression-Aware Content Protection against Agentic Crawlers

Source: arXiv:2607.08180 · Published 2026-07-09 · By Xuefei Wang

TL;DR

This paper addresses the emerging challenge of protecting high-value online textual content against unauthorized scraping and reuse by advanced LLM-based agentic crawlers which mimic human browsers to bypass classical access controls. Existing defenses either rely on pre-delivery measures that agents circumvent or inject visible perturbations that degrade human readability. The authors identify the routinely invoked context compression stage in agent workflows—where retrieved content is condensed to fit context windows—as a critical but overlooked chokepoint for defense. They propose CAPE, a novel framework that injects invisible perturbations into textual content which preserve human-readable appearance but induce maximal semantic degradation during agent compression.

CAPE operates in three stages: it first discovers transferable structural priors from a white-box surrogate compressor; next it applies prior-guided evolutionary search to adapt perturbations under query-limited black-box compressors; finally, it calibrates a preference ranker to prioritize candidate queries for efficient target adaptation. The method is evaluated on three content types (long-form text, code, dialogue histories) across four diverse compression targets (including GPT-4.1, Gemini 3 Flash, LangGraph, and GitHub Copilot). Results show CAPE outperforms strong baselines by up to 75.8% relative information loss increase while keeping the injected perturbations visually indistinguishable (average human-visible input difference ~1.4%). Real-world workflow tests demonstrate CAPE causes up to 59.7% accuracy drops in downstream tasks, confirming practical protection efficacy.

Key findings

• CAPE improves information degradation by up to 75.8% over the strongest baseline across three content types and four compression models.
• Invisible perturbations injected by CAPE cause severe semantic corruption after compression while maintaining human-visible difference (HVID) as low as 1.4%, drastically better than baselines causing visible distortions.
• Direct transfer of surrogate-derived perturbations without target adaptation yields significantly lower degradation (~33% TD) than full CAPE (~49% TD), showing adaptation is critical.
• On GPT-4.1 compression, CAPE achieves 49.2% textual degradation (TD) vs 17.4% for the next best baseline (HardCom).
• CAPE reduces LangGraph workflow accuracy by 59.7% (DRAD) while introducing only 2.8% input difference, outperforming HardCom (26.2% accuracy drop at 46.1% input diff).
• On GitHub Copilot code tasks, CAPE increases code semantic degradation (CSD) to 16.4 from 4.5 baseline, and morphological output error rate (MOR) to 38.5 from 2.1, with low visible difference.
• Ablations removing any core CAPE module (prior discovery, prior-guided evolution, preference calibration) reduce textual degradation by 7.5% to 26.7%.
• CAPE perturbations jointly optimized for multiple compressors still cause significant degradation on each target, demonstrating cross-model robustness.

Threat model

The adversary is a powerful agentic crawler deploying large language model agents that mimic human browsing behavior to retrieve online content. They possess unknown, closed-source compression modules configurable freely and aim to produce faithful compressed content views for unauthorized downstream reuse. The adversary receives protected content as published and cannot manipulate it at the source. They do not have prior knowledge of perturbations and do not aggressively sanitize content to remove invisible tokens, to preserve information completeness. The defender can only preprocess and modify content before publication, with white-box access to surrogate compressors but treats target compressors as opaque black boxes accessible through limited queries.

Methodology — deep read

The threat model assumes an adversary deploying LLM-powered agentic crawlers extracting online content without authorization, compressing and reusing it downstream. The adversary mimics human browsers and chooses unknown, closed-source compression modules. The defender is the content owner who can only preprocess input text before publication and has no access to the adversary’s models or parameters. The defender assumes white-box access only to surrogate compressors for perturbation design.

The defender's goal is to inject an invisible perturbation into the source content that is minimally different visually (measured by HVID) but causes maximal semantic degradation after compression (measured by Textual Degradation (TD), Information Degradation (ID), and Output Semantic Drift (OSD)). The perturbation consists of invisible Unicode tokens inserted at specific positions.

CAPE optimizes perturbations via a 3-stage pipeline: (1) Structural Prior Discovery uses a white-box surrogate compressor to optimize perturbations against a probing continuation, maximizing compression output entropy, promoting anomalous tokens, and suppressing natural continuation likelihood. This yields high-scoring seed perturbations from which structural priors are extracted—localized token fragments, co-occurrence patterns, and position-length compatibilities.

(2) Prior-Guided Evolutionary Adaptation performs discrete evolutionary search to adapt perturbations to black-box target compressors using surrogate priors to guide mutation and recombination. Candidate fitness combines measured degradation, prior consistency, stability, and diversity via regularized fitness scoring. A dynamic annealing mechanism balances exploration vs exploitation.

(3) Preference-Calibrated Query Selection trains a local ranker via pairwise preference losses on target compressor feedback to estimate degradation cheaply. It allocates limited compressor queries to candidates balancing exploitation, uncertainty, and structural novelty, iteratively refining the ranker and generating final perturbations.

Experiments use datasets of long-form text, code snippets, and dialogue histories from Task Haystack, CoRE, CAB, BABILong, and T1. Closed-source compressors GPT-4.1 and Gemini 3 Flash, LangGraph agent workflow, and GitHub Copilot are evaluation targets. Baselines include random invisible token insertion, fixed zero-width perturbations, TAP, HardCom, and CAPE ablated variants.

Metrics include textual degradation (TD), information degradation (ID), output semantic drift (OSD), human-visible input difference (HVID), workflow accuracy drop (DRAD), code semantic degradation (CSD), and malformed output rate (MOR).

The training regime involves grid search for hyperparameters, with 1/20 perturbation length ratio, and up to 100 target queries per example. Query budget constraints reflect realistic limited feedback from black-box compressors.

The evaluation protocol tests perturbations on held-out inputs and multiple compressors, reports absolute and relative metric improvements, and includes statistical significance via ablations. Real-world agent workflow impact is also measured.

Reproducibility is supported by public release of CAPE prototype and evaluation materials, although target compressors and datasets are mostly closed and proprietary.

Technical innovations

• Identification of context compression in LLM agent pipelines as a novel, orthogonal chokepoint for content protection beyond traditional access control.
• Design of CAPE framework combining distributional surrogate-based perturbation optimization with prior-guided evolutionary adaptation and preference-calibrated query selection to efficiently craft invisible tokens that degrade compression fidelity.
• Structural prior extraction capturing degradation-associated local fragments, co-occurrence patterns, and insertion position-length compatibility for transferable perturbation generalization.
• Closed-loop evolutionary search with dynamic annealing balancing exploration and exploitation, regularized by prior consistency and query feedback noise filtering.
• Preference ranker training from limited compressor queries using pairwise preference ranking loss to efficiently prioritize promising perturbation candidates under strict query budgets.

Datasets

• Task Haystack — unknown size — long-form documents
• CoRE — unknown size — code snippets
• CAB — unknown size — code snippets
• BABILong — unknown size — multi-turn dialogues
• T1 — unknown size — multi-turn dialogues

Baselines vs proposed

• HardCom: Textual Degradation (TD) = 17.4% (GPT-4.1 long-form text) vs CAPE: 49.2%
• HardCom: LangGraph downstream accuracy drop (DRAD) = 26.2% vs CAPE: 59.7%
• HardCom: GitHub Copilot code semantic degradation (CSD) = 4.5 vs CAPE: 16.4
• Random invisible perturbation: TD = ~0.5% vs CAPE: 49.2%
• Direct Transfer (no adaptation): TD = ~24.6% vs CAPE: 49.2%
• Prior-Free Evolution (no prior discovery): TD = ~32.5% vs CAPE: 49.2%
• TAP baseline: TD = 14.0% vs CAPE: 49.2%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.08180.

Fig 1: General agent workflow with compression mechanism.

Fig 2: Overview of CAPE. CAPE discovers structural priors from accessible compressors, adapts them through prior-guided

Fig 3 (page 5).

Fig 4 (page 5).

Fig 5 (page 5).

Fig 6 (page 5).

Fig 7 (page 5).

Limitations

• The evaluation uses known surrogate compressors and a limited set of black-box targets; the universality against arbitrary unknown compressors remains uncertain.
• Human-visible difference metrics rely on HVID and informal human judgments, but detailed user studies on human perception of perturbations are lacking.
• Code content protection is less effective than for free text due to syntax constraints, indicating limits on applicability to highly structured data.
• The method depends on injection of non-displayed tokens supported by Unicode and client rendering; environments that sanitize or strip such tokens may reduce efficacy.
• The query budget for target compressors is small (100 queries), so adaptation to highly restricted or no-query black-box models remains an open problem.
• The approach does not prevent original text copying or direct reuse if adversaries bypass compression and use raw content.

Open questions / follow-ons

• How effective are invisible perturbation defenses when adversaries apply active sanitization or normalization steps that remove zero-width or invisible Unicode tokens?
• Can CAPE's prior-guided evolutionary framework be extended to multimodal or non-textual content formats targeted by agentic crawlers?
• What are the robustness and transferability limits of CAPE against dynamically updating or adversarially trained compression modules?
• How do invisible perturbations interact with downstream agent modules beyond compression, such as memory writing or retrieval augmentation?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this work highlights a novel, complementary defense layer based on exploiting the compression stage in agent pipelines, rather than relying solely on access control or outwardly visible content obfuscation. CAPE's approach to injecting invisible perturbations allows content owners to proactively protect textual knowledge assets against advanced LLM-driven scrapers that bypass traditional bot detection techniques. The framework offers a method to degrade the compressed semantic representation agents rely on, thus limiting automated content reuse while preserving human-readability and user experience. Implementing similar defenses can serve as a fallback for cases when perimeter controls fail.

However, CAPE requires the ability to preprocess textual content prior to publication and relies on Unicode token injections supported by client rendering environments, so bot-defense teams should consider compatibility with their content delivery platforms and adversary sanitization capabilities. This paper informs future CAPTCHA and bot defense research by demonstrating the potential of internal, inference-time disruptions in agent ingestion pipelines as a stealthy and effective anti-scraping strategy that does not depend on external interaction challenges or user friction.

Cite

@article{arxiv2607_08180,
  title={ Out of Sight: Compression-Aware Content Protection against Agentic Crawlers },
  author={ Xuefei Wang },
  journal={arXiv preprint arXiv:2607.08180},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.08180}
}

Read the full paper

• arXiv:2607.08180 (abstract)
• arXiv:2607.08180 (PDF)