VESTIGE: A Knowledge-Guided Masking Strategy for Corruption-Aware Fine-Tuning of Genomic Transformers, Validated on Ancient DNA Reconstruction

Source: arXiv:2607.27712 · Published 2026-07-30 · By Angshuman Chakravertty, Rahul Maheshwari

TL;DR

This paper addresses a fundamental limitation in standard masked language model (MLM) fine-tuning applied to genomic sequences, particularly ancient DNA (aDNA). Traditional MLM applies a uniform 15% masking probability across all sequence positions, implicitly assuming that reconstruction difficulty is uniform. However, biological degradation processes like cytosine deamination in aDNA are sharply position-dependent, predominantly impacting nucleotides near fragment ends, making uniform masking suboptimal or even harmful. The authors present VESTIGE, a corruption-aware masking collator that redistributes the masking probability across sequence positions proportional to empirically measured position-specific damage rates derived from mapDamage2 profiles. This produces a parameter-free, plug-in replacement for MLM masking that requires no model or loss changes.

They validate VESTIGE on woolly mammoth aDNA coding sequences from two specimens across seven genes using DNABERT-2, showing consistent and statistically robust gains in reconstruction accuracy. Compared to uniform MLM masking, VESTIGE improves nucleotide recovery rates by 4-10 percentage points across all tested terminal-zone widths, reduces validation cross-entropy by 13%, and yields highly plausible protein structure predictions with TM-scores > 0.95 by ESMFold even under damage rates amplified 10-30× beyond biological levels. A biosecurity classifier flags <2% of windows, all attributable to reference-genome features rather than reconstruction artefacts. The authors further argue the approach is broadly extensible to other genomics degradation modalities and sequencing error profiles. In summary, VESTIGE demonstrates that knowledge-guided masking aligned to empirical corruption profiles sharply improves transformer fine-tuning for degraded genomic sequences.

Key findings

• VESTIGE outperforms standard uniform MLM masking in nucleotide recovery rate (NRR) by +4.18 to +10.35 percentage points across six terminal-zone widths (3 to 25 positions near fragment ends), all with p < 10^-8 over 626 paired evaluation windows.
• Validation cross-entropy is reduced by 13% with VESTIGE versus standard MLM (3.274 vs. 3.757).
• At the innermost terminal zone (position 1), standard MLM recovers only 20.49% of damaged nucleotides, below a frequency-matched random baseline of 27.67%, whereas VESTIGE achieves 30.84% recovery (p = 1.33 × 10^-10, n=486).
• ESMFold structural reconstructions of sequences reconstructed by VESTIGE achieve TM-score > 0.95 across six reconstructions (three genes), even when post-mortem damage (PMD) rates are artificially amplified 10–30×.
• A 1D CNN biosecurity classifier returns AUC = 0.935 on held-out virulence-gene test data and clears 98.2% of reconstructed windows, with the 1.76% flagged due to reference genome features, not reconstruction errors.
• Masking probabilities are position-and-base dependent, with C and G positions near fragment termini receiving up to ~28% masking after rescaling, versus uniform 15% in standard MLM.
• Use of empirical damage profiles from mapDamage2 provides a domain-agnostic corruption prior, enabling corruption-aware masking for various degraded sequence contexts beyond aDNA.
• The masking collator substitution alone, with model, data, seed, and hyperparameters fixed, accounts for all observed improvements, isolating spatial masking redistribution as the sole variable.

Methodology — deep read

1. Threat Model & Assumptions: The adversary scenario is not explicitly security-focused; rather, the 'corruption' is a domain-specific, endogenous biological degradation process—cytosine deamination in aDNA producing position-dependent C→T and G→A substitutions concentrated near fragment ends. The model assumes availability of an empirical per-position damage profile (derived externally via mapDamage2) that is fixed and known at fine-tuning time. Adversary capabilities are irrelevant as this focuses on reconstruction accuracy over naturally occurring damage patterns.

2. Data: The study uses ancient DNA from two woolly mammoth specimens, dated ~4,300 and ~44,800 years BP, obtained from public Sequence Read Archive project ERP008929. Raw FASTQ reads were subsampled to 5 million reads each, aligned to the African elephant reference genome (Elephas maximus GCF_024166365.1) using BWA-aln with ancient DNA-tuned parameters disabling seeding and allowing mismatch rates matching deamination. Per-position damage profiles were obtained via mapDamage2, producing arrays of C→T frequencies at 5' ends and G→A frequencies at 3' ends, decaying exponentially into the fragment interior over about 70 positions.

Coding sequences (CDS) for three cold-adaptation genes (TRPV3, KCNK9, HBB) were extracted and tiled into 2,000 bp sliding windows with 200 bp stride, tokenized with DNABERT-2’s BPE tokenizer max 512 tokens. This generated 344 windows, split 80/20 into 275 training and 69 validation windows identified by unique gene coordinates. Four additional held-out genes (TRPA1, UCP1, ADRB3, FASN) were reserved for evaluation to test generalization.

3. Architecture / Algorithm: The backbone model is DNABERT-2, a 117M parameter BERT-style transformer encoder pre-trained with BPE tokenization on multispecies genomes, unchanged in weights and architecture during fine-tuning. The only modification is the replacement of the standard MLM collator with VESTIGE’s DamageAwareDataCollator, which uses empirical damage profiles to assign position-specific masking probabilities only to C and G tokens (masking A/T probabilities fixed at zero).

Masking probability at position p for base b is proportional to the mean C→T or G→A damage frequency at that position (from the mapDamage2 arrays, averaged across specimens). To maintain fairness, this positional masking probability distribution is linearly rescaled so that the mean masking rate over all C/G positions equals 15%, matching the standard MLM masking density overall, thereby isolating only the spatial redistribution of masking as a variable.

Masking replacement follows BERT’s 80/10/10 scheme: 80% masked with [MASK], 10% substituted with random tokens, 10% left unchanged.

4. Training Regime: Fine-tuning was conducted for 20 epochs with batch size 8, learning rate 2e-5, using cosine learning rate schedule with 10% warmup, weight decay 0.01, and fixed random seed 42. Runs were performed on Apple M1 Pro with MPS backend via HuggingFace Trainer. Both VESTIGE and baseline MLM fine-tuning runs shared identical setup and differed only in data collator masking distribution. Training wall times were 11h24m for MLM and 14h32m for VESTIGE due to additional computation in collator.

5. Evaluation Protocol: Evaluation uses held-out validation sets and separate held-out genes for generalization. The main metric is nucleotide recovery rate (NRR), the fraction of masked positions where the predicted token exactly matches ground truth. Evaluation is done for varying terminal-zone widths (3,5,10,15,20,25 nucleotides near fragment ends). Statistical significance is assessed via paired analyses over 626 paired windows, with p-values < 10^-8 reported.

Validation cross-entropy is measured on validation set.

Structural validity of reconstructed coding sequences is assessed by ESMFold prediction, with superposition and TM-score calculation against an elephant reference structure, evaluating biological plausibility.

Biosecurity evaluation is performed using a 1D CNN classifier trained on 11 virulence gene classes, calculating AUC and the proportion of reconstructed windows flagged as potential biosecurity risks.

6. Reproducibility: Code implementing VESTIGE’s masking collator and evaluation pipelines was logged with Weights & Biases. Raw data sources are public SRA datasets with accession details provided. Damage profiles are derived with public mapDamage2 software. Details on tokenizer offset mapping are provided to ensure precise masking and evaluation targeting only empirically damaged nucleotides. However, frozen weights and full source code release details are not explicitly stated, but logging hints at reasonable reproducibility.

Technical innovations

• Introduction of VESTIGE, a parameter-free corruption-aware masking collator that aligns MLM mask probabilities with empirical per-position damage profiles from biological degradation rather than uniform masking.
• Use of mapDamage2-generated position-specific cytosine deamination frequency profiles to guide masking selectively on C and G nucleotides, rescaled to maintain standard 15% masking density.
• Isolating spatial redistribution of masking probabilities as the sole experimental variable through controlled ablation using identical model weights, data, seed, and hyperparameters.
• Demonstration that knowledge-guided masking substantially improves reconstruction accuracy and downstream structural and biosecurity validation without any architectural changes or additional parameters.

Datasets

• Woolly mammoth CDS dataset — 344 sliding windows — public SRA project ERP008929
• Elephas maximus reference genome (GCF_024166365.1) — full genome — public
• Held-out virulence gene validation set for biosecurity classifier — unspecified size — public (Park et al., 2023)

Baselines vs proposed

• Standard MLM masked language model: nucleotide recovery rate = 20.49% at terminal zone width 3 vs VESTIGE: 30.84% (p=1.33 × 10^-10, n=486)
• Baseline MLM cross-entropy = 3.757 vs VESTIGE cross-entropy = 3.274 (13% reduction)
• NRR gain across terminal zones [3,5,10,15,20,25]: +4.18 to +10.35 percentage points (all p < 10^-8, n=626 paired windows)
• ESMFold TM-score on protein structure prediction: baseline not explicitly stated vs VESTIGE > 0.95 across all six reconstructed proteins even under 10–30× amplified damage rates
• 1D CNN biosecurity classifier AUC = 0.935 on held-out virulence gene set; 98.2% reconstructed windows cleared vs 1.76% flagged attributable to reference genome features, no difference in reconstruction method impact

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.27712.

Fig 1: The VESTIGE end-to-end pipeline - the single methodological substitution is

Fig 2: Post-mortem damage profiles (C→T and G→A) for both woolly mammoth

Fig 3: DAM lowers validation cross-entropy throughout fine-tuning, evidencing that

Fig 4: Standard MLM produces a worse-than-random model at the very positions that

Fig 5: DAM’s lead over MLM spans all six terminal-zone widths and is largest at

Fig 6: shows the DAM advantage is sharpest at 𝑑= 1 where DAM recovers 31.25% against

Fig 7: NRR vs. simulated peak PMD rate (5%–40%). DAM advantage is significant at

Fig 8: ESMFold reconstructions achieve TM-score > 0.95 across all six reconstructions

Limitations

• Dataset limited to only two ancient specimens from a single species (woolly mammoth), restricting generalizability across taxa and damage profiles.
• Fine-tuning and evaluation limited to a small gene set (3 training genes, 4 held-out genes), with relatively few total windows (344), so scalability to whole genomes untested.
• Damage profiles only derived from mapDamage2 for cytosine deamination; other aDNA damage types or sequencing platform errors less explored.
• No adversarial or out-of-distribution degradation attacks conducted to test robustness beyond empirical profiles.
• Limited discussion of potential impact on longer reads or differing fragmentation patterns, common in other ancient DNA datasets.
• Biosecurity classifier validation is indirect—no explicit testing of reconstructed pathogenic sequences to assess false negatives.

Open questions / follow-ons

• How well does VESTIGE generalize to larger, more diverse ancient DNA datasets spanning multiple species, preservation conditions, and damage types?
• Can the corruption-aware masking scheme improve reconstruction for non-aDNA degraded sequences, such as formalin-fixed paraffin-embedded (FFPE) samples or bisulfite-treated DNA, in practice?
• What impact does corruption-aware masking have on transformer models trained from scratch rather than fine-tuned from pre-trained weights?
• How might curriculum learning or adaptive masking schedules further optimize spatial masking distributions beyond static empirical priors?

Why it matters for bot defense

For practitioners developing bot defense or CAPTCHA systems reliant on sequence modeling, VESTIGE exemplifies the importance of aligning training data masking strategies with domain-specific corruption or noise profiles. While CAPTCHAs typically do not involve genomic data, the principle of corruption-aware masking could inspire training routines for transformers processing other noisy or degraded sequential inputs such as user input logs, behavioral telemetry, or sensor data. Instead of uniformly masking tokens, incorporating knowledge of where noise or adversarial manipulations concentrate can help models learn more robust, informative representations. For bot defense, masking corrupt or uncertain tokens at higher rates can guide models to focus learning capacity on the most challenging, attack-relevant portions of sequences. Additionally, the approach underscores the utility of lightweight data collator modifications without changing model architecture, which could ease integration. However, rigorous evaluation of how domain knowledge about corruption patterns transfers across modalities remains essential.

Cite

@article{arxiv2607_27712,
  title={ VESTIGE: A Knowledge-Guided Masking Strategy for Corruption-Aware Fine-Tuning of Genomic Transformers, Validated on Ancient DNA Reconstruction },
  author={ Angshuman Chakravertty and Rahul Maheshwari },
  journal={arXiv preprint arXiv:2607.27712},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.27712}
}

Read the full paper

• arXiv:2607.27712 (abstract)
• arXiv:2607.27712 (PDF)