Video Transformer for Remote Identity Document Hologram Detection

Source: arXiv:2607.11419 · Published 2026-07-13 · By Joris Voerman, Nicolas Sidere, Jean-Christophe Burie

TL;DR

The paper addresses the challenge of remote identity document verification with a focus on detecting holograms—a key anti-counterfeiting security feature—in videos captured by consumer smartphones. Existing methods struggle due to limited availability of robust datasets, the need for video-based analysis to recognize subtle holographic light effects, and constraints on computational resources for deployment. To tackle these, the authors propose ViTransHolo, a lightweight video transformer model adapted from a face presentation attack detection architecture, designed to work efficiently on medium- to small-sized datasets while maintaining high accuracy.

ViTransHolo leverages EfficientNet as a convolutional encoder and a stack of transformer layers enhanced with multi-scale multi-head self-attention to capture spatial-temporal dependencies in ID videos. Through comprehensive evaluation on the public MIDV-Holo dataset and a private real-world IML3 dataset of French IDs, the method achieves substantial improvements over prior state-of-the-art baselines, notably a +26.86% recall and +17.93% accuracy gain compared to the best previous MIDV-Holo baseline. Further experiments demonstrate strong generalization across challenging fraud scenarios (including unseen photo replacement attacks), robustness to limited training data, and favorable computational efficiency. The authors integrate ViTransHolo into a broader remote ID verification system architecture, highlighting its practical applicability and potential for real-world deployment.

Key findings

• ViTransHolo achieves 98.43% accuracy on MIDV-Holo, improving +17.93% over the best MIDV-Holo baseline accuracy of 80.50%.
• Recall on MIDV-Holo improved +26.86%, reaching 98.19% versus 71.33% for the prior baseline.
• False positive rate dropped from 10.33% to 1.34%, a -7.33% absolute reduction compared to MIDV-Holo baseline.
• On the private IML3 dataset of real French IDs, ViTransHolo reaches 100% across all metrics (FPR, Recall, Accuracy, F1-score), though authors note dataset size is small (17 documents).
• Model generalizes well to unseen photo replacement fraud (held-out during training), maintaining >90% accuracy in that scenario.
• Reducing training data from 70% to 30% of samples causes only a minor accuracy drop (~7%), showing data efficiency.
• Layer ablation study shows that reducing transformer layers from 6 to 1 cuts inference time by ~40% with <2% accuracy loss.
• Inference speed is on the order of hundreds of milliseconds, substantially faster than MIDV-Holo baseline which runs in seconds.

Threat model

The adversary is a fraudster equipped with typical forgery tools aiming to create counterfeit identity documents without specialized hardware. Attacks considered include removal or imitation of holograms by photocopying, pseudo-holograms, or photo replacement patches. The adversary cannot deploy methods requiring special sensors (infrared, hyperspectral imaging) or generate perfect authentic hologram replicas. The system assumes video input from standard consumer smartphones without hardware-based security augmentation.

Methodology — deep read

1. Threat model & assumptions: The adversary is a fraudster who attempts to create counterfeit identity documents with fake or no holograms, including photocopied, pseudo-holograms, or photo replacement attacks. Fraud types requiring specialized hardware or hyperspectral imaging are excluded since only consumer smartphone video is assumed. The system assumes normal smartphone videos as input and does not rely on infrared or special sensors.

2. Data: Two datasets are used. The public MIDV-Holo dataset includes 300 "original" and 400 "fraud" videos of mock ID cards and passports under varying light conditions, with four classes of fraud: no hologram, photocopied hologram, pseudo-hologram, and photo replacement. The private IML3 dataset contains 748 video clips from 17 real French ID documents across multiple document types and smartphone models, split 50/50 between originals and fakes (reprints). Train/test splits avoid leakage by grouping videos by document. Preprocessing extracts and crops document regions resized to 224x224.

3. Architecture: Input videos are uniformly sampled to 50 frames, each cropped and resized. Frames pass through EfficientNet convolutional encoder producing spatial features. These feed into a stack of six transformer layers equipped with Multi-Scale Multi-Head Self-Attention (MS-MHSA) to capture spatial-temporal dependencies. Each transformer layer contains attention and convolutional feed-forward blocks with ReLU activations. A final MLP classifies each frame; video-level prediction averages frame outputs.

4. Training regime: Binary cross-entropy loss with Adam optimizer and cosine annealing warm restart scheduler is used. Experiments run on NVIDIA A40 GPUs. Each training is repeated 5 times with different random seeds. The model is trained on 70% of data with 15% validation and 15% test, ensuring no leaks across splits at document level.

5. Evaluation protocol: Metrics are False Positive Rate (FPR), Recall, Standard Accuracy, and F1-score. ViTransHolo is compared to MIDV-Holo baseline and a triplet-loss model. Experiments include fraud type-specific evaluation, training data size ablation, and transformer layer reductions. Photo replacement fraud samples are kept exclusively for testing to evaluate out-of-distribution generalization.

6. Reproducibility: Implementation is in PyTorch. The private IML3 dataset is not public due to privacy, limiting exact reproductions. MIDV-Holo dataset is public. Model details and training protocol are described sufficiently for academic replication. No mention of public code release or pretrained weights.

Example end-to-end: A 50-frame video of a French ID captured by smartphone is preprocessed by cropping the document region and resizing. Frames pass through EfficientNet to produce features that enter transformer layers implementing MS-MHSA attention. Outputs are classified on a per-frame basis, then averaged for video-level decision. This handles various fraud types by learning subtle temporal-spatial hologram characteristics, improving recall and lowering false positives compared to baselines.

Technical innovations

• Adaptation of ViTransPAD video transformer architecture with EfficientNet encoder and six MS-MHSA equipped transformer layers tailored for hologram detection in ID videos.
• Use of multi-scale multi-head self-attention (MS-MHSA) to capture spatial-temporal dependencies crucial for detecting holograms' unique light reflections in video frames.
• Integration of convolutional feed-forward layers with ReLU within each transformer layer to improve spatial feature extraction.
• Frugality analysis demonstrating data efficiency and computational-cost trade-offs via training data reduction and transformer layer ablation studies.

Datasets

• MIDV-Holo — 700 videos total (300 genuine + 400 fraud) — public dataset of mock ID documents from fictional country
• IML3 — 748 video clips from 17 real French identity documents — private dataset from La Rochelle University

Baselines vs proposed

• MIDV-Holo baseline: Accuracy = 80.50% vs ViTransHolo: 98.43% accuracy
• MIDV-Holo baseline: Recall = 71.33% vs ViTransHolo: 98.19% recall
• MIDV-Holo baseline: FPR = 10.33% vs ViTransHolo: 1.34% FPR
• Triplet-loss baseline: F1-score = 90% vs ViTransHolo: 98.41% F1-score
• ViTransHolo inference time: hundreds of milliseconds vs MIDV-Holo baseline: seconds

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.11419.

Fig 1: Overall diagram of the proposed system for remote ID authentication. The colors correspond to specifics sub-systems : yellow for classification,

Fig 2: ViTransHolo model architecture. It begin with a embedding process to format input video, then an encoder (EfficientNet), followed by six

Fig 3: Samples of each MIDV-Holo challenges.

Fig 4: MIDV-Holo hologram templates.

Fig 5: Document classes represented in IML3. These official specimen

Limitations

• Private IML3 dataset is small and focused solely on French IDs, limiting geographic and document diversity.
• MIDV-Holo dataset hologram templates are similar and feature limited document classes (ID cards, passports) which may limit generalization.
• Model not tested against adversarial attacks beyond the defined fraud types; certified robustness to adaptive adversaries is unknown.
• Photo replacement fraud samples were not used for training; while generalization is strong, testing on additional unseen fraud types is needed.
• No ablation on number of input frames or video resolution; sensitivity to video quality variations remains unexplored.
• No public release of code or pretrained models to facilitate replication.

Open questions / follow-ons

• Can the hologram detection model be adapted or extended to reconstruct hologram 3D structure for more fine-grained forgery detection?
• How well does ViTransHolo perform under adverse video capture conditions like motion blur, extreme lighting, or very low resolution?
• Does the model maintain robustness against advanced adversarial manipulations designed to fool hologram detectors (e.g., AI-generated spoofing)?
• What are the optimal trade-offs in transformer layer depth and frame sampling for deployment on resource-constrained mobile devices?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this work illustrates the application of video transformer architectures to detect subtle, authentic visual security elements under constrained data and compute resources. The approach highlights how spatial-temporal attention mechanisms can robustly distinguish genuine holographic effects from forgeries using only smartphone video. This is relevant for multi-factor verification systems incorporating physical document features as liveness and authenticity cues.

Additionally, the frugality experiments provide a valuable example of balancing performance with computational cost, an important consideration for real-time verification. The demonstrated generalization to unseen fraud types also suggests the potential of video transformers in detecting adaptive attacks. These insights can inform defenses beyond CAPTCHAs, especially for identity verification, fraud detection, or bot-prevention systems that leverage physical-world signals in constrained, mobile-first contexts.

Cite

@article{arxiv2607_11419,
  title={ Video Transformer for Remote Identity Document Hologram Detection },
  author={ Joris Voerman and Nicolas Sidere and Jean-Christophe Burie },
  journal={arXiv preprint arXiv:2607.11419},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.11419}
}

Read the full paper

• arXiv:2607.11419 (abstract)
• arXiv:2607.11419 (PDF)