Open-Set Visual Text Forensics via Sparse-Constraint Rectified Flow

Source: arXiv:2608.02258 · Published 2026-08-03 · By Jiangling Zhang, Shuxuan Gao, Zeyu Chen, Yichao Liu, Yu Zhou

TL;DR

This paper addresses the limitations of current visual text tampering detectors in handling open-set and unseen forgery patterns enabled by evolving generative AI. Traditional discriminative detectors tend to overfit specific manipulation artifacts, hindering generalization to novel tampering techniques. To improve open-set robustness, the authors propose a generative detection framework that models authenticity as an empirical distribution of natural visual-text statistics and estimates the local restoration cost needed to align a tampered image back to this authentic distribution. The core technical innovation is Sparse-Constraint Rectified Flow (SC-RF), a flow-matching based method that learns a vector field representing local restoration velocity, emphasizing sparse tampered areas via a spatially weighted loss. They further integrate Forensic-DiT, a pixel-space diffusion transformer architecture designed to preserve high-frequency forensic traces via multimodal inputs (RGB, SRM residuals, DCT features) and a frequency-gated attention mechanism.

Experimentally, the method outperforms multiple state-of-the-art baselines on three benchmarks (Tampered-IC13, DocTamper, OSTF) in both zero-shot and fully supervised fine-tuning modes, displaying superior F1-score and IoU localization metrics by margins of 3.2 and 4.8 points on average, respectively. The detector also exhibits strong zero-shot generalization to unseen text editing types. An auxiliary stress-test demonstrates that applying their model’s local harmonization reduces cues exploited by existing detectors, indicating new complementary vulnerabilities and highlighting the importance of generative alignment models for open-set forensic robustness.

Key findings

• Proposed SC-RF method surpasses runner-up methods by an average of 3.2 F1 points and 4.8 IoU points on three benchmarks (Table 1).
• Zero-shot performance of SC-RF on unseen text editing patterns in OSTF benchmark achieves F1 = 0.781 and IoU = 0.641, close to fully supervised results.
• Training with self-supervised Artifact Injection and three-stage curriculum facilitates learning restoration-cost vector field without paired forged samples.
• Multi-modal Forensic Tokenizer (RGB+SRM+DCT) enhances capturing microscopic forgery traces.
• Frequency-Gated Multi-Head Self-Attention reduces incorrect context blending, improving localization consistency.
• Stress-test shows that applying the model’s local harmonization leads to substantial F1 score drops (e.g., −0.260 average for DTD forensic detector) in existing tampering detection models (Table 2).
• t-SNE visualization reveals post-harmonization tampered samples shift toward authentic cluster in feature space, not just random perturbation (Fig 4).
• Sparse-Constraint weighting with λ=20 mitigates optimization bias from vast authentic background regions to focus learning on sparse tampering regions.

Threat model

The adversary is a forger who applies sparse visual text tampering manipulations on natural scene or document images to evade detection. They can create locally edited text regions with subtle anomalies such as interpolation inconsistencies, compression artifacts, or noise mismatches but cannot fully control the underlying physical formation statistics. The attacker is unknown in terms of forgery patterns, aiming to evade detectors that rely on closed-set artifact distributions. The defender’s goal is to identify local deviations from authentic visual-text statistical distributions without prior knowledge of forgery specifics.

Methodology — deep read

1. Threat model & assumptions: The adversary is a forgery operator who applies unknown text editing/tampering to an image, producing sparse, localized anomalies. The model assumes no paired forged input-output supervision for training and no prior knowledge of specific forging techniques; the goal is open-set detection without overfitting to known forgery artifacts.

2. Data: The training corpus combines diverse authentic visual text images from seven scene text datasets (LSVT, ReCTS, ICDAR2013/2015/2017, TextOCR, ArT) plus RVL-CDIP document images to cover natural scenes and scanned/phographed documents. All training data are authentic (non-forged). Evaluation is performed on three benchmarks with tampered images: Tampered-IC13, DocTamper, and OSTF (the latter emphasizing open-set forgery diversity).

3. Architecture/Algorithm: They adapt Rectified Flow Matching to learn a time-dependent vector field v_θ that transports spatially localized corrupted images x_0 to authentic images x_1 along a straight-line geodesic interpolation x_t = (1–t)x_0 + t x_1. The velocity vector field is predicted to approximate u = x_1 – x_0 at each pixel, whose magnitude indicates restoration cost (local anomaly).

Sparse tampering results in most pixels being authentic, causing trivial solutions when optimizing standard flow matching. To address this, they propose a Sparse-Constraint Rectified Flow (SC-RF) objective that weights the loss spatially with W_s = 1 + λ·M_s, where M_s is the tampering mask and λ ≫ 1 (set to 20). This amplifies gradients on tampered pixels during training.

The specialized Forensic-DiT model is based on U-DiT transformer with U-Net style skip connections. It replaces standard patch tokenizer with a multi-modal forensic tokenizer that extracts combined embeddings from RGB image patches, SRM filter residuals, and Block-DCT frequency coefficients. Scale and shift parameters in layer norm are adaptively adjusted by a Self-Fingerprint Adaptive Normalization (SF-AdaLN) module that uses a global forensic fingerprint pooled from tokens.

A Frequency-Gated Multi-Head Self-Attention uses a learned frequency bias matrix computed from differences between frequency tokens, suppressing attention between spectrally dissimilar patches to prevent semantic blending of tampered/background areas.

4. Training regimen: Without paired forged data, they synthesize local micro-artifact corruptions via Artifact Injection applying Gaussian blur, JPEG compression, noise, alpha blending mismatches, and glyph re-rendering locally on authentic images. A three-stage curriculum decreases the artifact spatial coverage from 50-80% in warm-up to 1-5% realistic sparsity in refinement over 300k iterations with batch size 16.

AdamW optimizer with initial lr=1e-4, weight decay=1e-4, cosine annealing with linear warmup for 5k steps, and EMA with decay 0.9999 are used. Training runs on 4 NVIDIA RTX A6000 GPUs.

5. Evaluation protocol: Evaluation uses pixel-level F1-score, Intersection over Union (IoU), and Area Under ROC Curve (AUC) on benchmark test sets. Zero-shot tests directly apply the model trained only on authentic data. Full-shot fine-tuning evaluates benefits of supervised adaptation. Fixed score thresholds per dataset are used to convert restoration cost maps into binary masks.

6. Reproducibility: Code or pretrained weights are not explicitly mentioned as released. Dataset use relies on public or community-recognized datasets, but extensive retraining of baselines and ablations are conducted to ensure fair comparison.

A concrete example: Given an authentic image x_1 with sparse tampered regions M, they generate corrupted sample x_0 by injecting artifacts locally. The model learns to predict the flow velocity vector field v_θ(x_t, t) approximating x_1 – x_0, weighted by sparse mask, so that at inference on tampered query x_q, the instantaneous predicted vector magnitude ∥v_θ(x_q,0)∥ highlights tampered pixels, enabling localization via thresholding.

Technical innovations

• Sparse-Constraint Rectified Flow (SC-RF): a loss reweighting scheme that enforces flow matching optimization focus on spatially sparse tampered regions by weighting anomalies with a large λ factor.
• Forensic-DiT architecture combining multi-modal patch embeddings (RGB, SRM residuals, Block-DCT) to capture high-frequency forensic traces in pixel space without VAE compression.
• Self-Fingerprint Adaptive Normalization (SF-AdaLN) that modulates normalization parameters using a global forensic fingerprint extracted per image to adaptively account for domain and acquisition noise variations.
• Frequency-Gated Multi-Head Self-Attention: attention mechanism biased by frequency-domain differences to restrict cross-token attention between patches with distinct spectral statistics, preserving anomaly localization.

Datasets

• LSVT — tens of thousands images — public scene text dataset
• ReCTS — thousands of images — public scene text dataset
• ICDAR2013/2015/2017 — hundreds to thousands images each — public scene text datasets
• TextOCR — thousands of images — public scene text dataset
• ArT — public scene text dataset
• RVL-CDIP — over 400k document images — public document image dataset
• Tampered-IC13 (T-IC13) — hundreds of images with tampering — canonical scene text tampering benchmark
• DocTamper — large-scale document image tampering dataset
• OSTF — open-set text forgery dataset with diverse unseen manipulations

Baselines vs proposed

• MVSS-Net (full-shot): Avg F1 = 0.446, IoU = 0.290 vs Ours (zero-shot): F1 = 0.809, IoU = 0.685
• PSCC-Net (full-shot): Avg F1 = 0.401, IoU = 0.254 vs Ours (full-shot): F1 = 0.869, IoU = 0.773
• DeepLabV3+ (full-shot): Avg F1 = 0.630, IoU = 0.468 vs Ours (full-shot): F1 = 0.869, IoU = 0.773
• SegFormer (full-shot): Avg F1 = 0.638, IoU = 0.479 vs Ours (full-shot): F1 = 0.869, IoU = 0.773
• DTD (full-shot): Avg F1 = 0.775, IoU = 0.647 vs Ours (full-shot): F1 = 0.869, IoU = 0.773
• DAF (full-shot): Avg F1 = 0.814, IoU = 0.695 vs Ours (full-shot): F1 = 0.869, IoU = 0.773
• TTDMamba (full-shot): Avg F1 = 0.837, IoU = 0.725 vs Ours (full-shot): F1 = 0.869, IoU = 0.773
• Stress-test: DTD F1 drops by 0.260 on average after input processed by F-DiT local harmonization

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.02258.

Fig 2: Overview of our proposed framework. (a) Data Preparation: We synthesize the source distribution 𝑝0 via self-supervised

Fig 3: Qualitative comparison of tampering localization results on challenging samples from OSTF, DocTamper, and T-IC13

Fig 3 (page 4).

Fig 4: t-SNE visualization of feature distributions. (a) In

Fig 5 (page 6).

Fig 6 (page 6).

Fig 7 (page 6).

Fig 8 (page 6).

Limitations

• No explicit adversarial attack evaluation on highly adaptive forgery or evasive forgeries was conducted; robustness under worst-case targeted attacks remains untested.
• The method relies on self-supervised artifact injection for proxy tampering synthesis, which may not fully capture all real-world forgery modes.
• Large training compute and resources are needed due to transformer backbone and high-resolution pixel-space modeling.
• No mention of inference speed or real-time applicability; flow matching prediction/vector field estimation computational cost is not detailed.
• Some hyperparameters such as sparsity weight λ and frequency gate scaling η are empirically chosen; sensitivity analysis is limited.
• Code and weights release status not mentioned, so full reproducibility by third parties may be limited.

Open questions / follow-ons

• Can the Sparse-Constraint Rectified Flow framework be extended to handle video sequences with temporal consistency constraints?
• How does the approach perform against more adaptive adversarial forgers who optimize manipulations specifically to minimize restoration vector magnitudes?
• Can the localized restoration-cost maps be used to guide automatic forgery correction or inpainting beyond detection?
• What is the impact of different forensic feature extractors beyond SRM and DCT (e.g., learned noise residuals) in the multi-modal tokenizer?

Why it matters for bot defense

This work introduces a fundamentally different approach for visual text forgery detection, shifting from discriminative pattern recognition to a generative restoration-cost estimation paradigm. For bot-defense engineers and CAPTCHA practitioners, this suggests the value in modeling authenticity distributions and detecting anomalies as deviations from these, rather than training classifiers on known manipulation artifacts that become obsolete as forgery techniques evolve. The Sparse-Constraint Rectified Flow method offers a more robust open-set detection capability that can generalize to unseen text manipulations common in AI-powered attacks.

The Forensic-DiT architecture's design considerations—such as preserving high-frequency forensic signals and frequency-gated attention—highlight important insights about the kinds of subtle statistical inconsistencies that open-set tampering detection must exploit. The auxiliary stress-test analysis also provides a cautionary perspective: local harmonization or restoration-oriented manipulations may reduce detectability by existing forensic models, indicating the need for future detectors to incorporate generative inconsistencies or restoration-cost features as complementary signals. Overall, adopting flow-based generative anomaly detection may enhance resilience of text and visual CAPTCHA schemes against sophisticated AI-driven manipulations.

Cite

@article{arxiv2608_02258,
  title={ Open-Set Visual Text Forensics via Sparse-Constraint Rectified Flow },
  author={ Jiangling Zhang and Shuxuan Gao and Zeyu Chen and Yichao Liu and Yu Zhou },
  journal={arXiv preprint arXiv:2608.02258},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.02258}
}

Read the full paper

• arXiv:2608.02258 (abstract)
• arXiv:2608.02258 (PDF)