VQ-VAD: Vector-quantized Motion Representation Learning for Human-centric Video Anomaly Detection

Source: arXiv:2608.05069 · Published 2026-08-05 · By Narges Rashvand, Ghazal Alinezhad Noghre, Shanle Yao, Gabriel Maldonado, Hamed Tabkhi

TL;DR

Video Anomaly Detection (VAD) is challenging due to the scarcity of anomalous data and significant visual variability in surveillance footage. To address privacy and noise concerns, pose-based VAD methods focus on human motion rather than raw pixels. However, existing pose-based approaches typically use continuous latent spaces, which limit their ability to learn compact and transferable motion representations crucial for robust anomaly detection. This paper proposes VQ-VAD, a novel human-centric VAD framework that learns discrete motion representations via vector quantization. Drawing inspiration from VQ-GANs originally developed for image synthesis, VQ-VAD adapts this paradigm to sequences of human keypoints, building a discrete codebook of normal motion patterns. Anomalies are detected as sequences that reconstruct poorly through this learned vocabulary.

Extensive experiments on four standard benchmarks demonstrate that VQ-VAD achieves strong in-domain anomaly detection performance (e.g., 81.83% AUC on HR-SHT) and robust cross-domain generalization (up to 76.69% AUC on HR-SHT when trained on CMU Panoptic). The discrete motion vocabulary learned enables effective generalization across datasets with diverse visual contexts, outperforming or matching prior pose-based continuous latent space methods. These results illustrate the promise of vector-quantized discrete embeddings for more interpretable, compact, and robust behavior modeling in human-centric video anomaly detection.

Key findings

• VQ-VAD achieves 81.83% AUC-ROC in in-domain anomaly detection on HR-SHT dataset, competitive with state-of-the-art.
• Cross-domain evaluation with training on CMU Panoptic and testing on HR-SHT yields 76.69% AUC-ROC without retraining.
• Cross-dataset generalization shows VQ-VAD outperforms STG-NF by over 5% on HuVAD dataset (59.66% vs 54.32% AUC).
• Best reconstruction obtained at temporal compression factor F4 and codebook size 1024 (MPJPE 0.0722, L1 error 0.0459).
• Increasing codebook size from 256 to 1024 consistently improves reconstruction quality and motion token richness.
• VQ-VAD’s anomaly score is the reconstruction error of pose sequences quantized through the learned discrete codebook.
• Discrete latent representation provides structured, compact motion vocabulary improving robustness to domain shifts.
• Adversarial training alongside reconstruction and vector-quantization losses improves motion realism and representation stability.

Threat model

The adversary generates abnormal human motion patterns that differ from the normal behavior used in training, but cannot interfere with or manipulate the pose estimation process. The model is trained only on normal motion sequences and detects anomalies as motion sequences that cannot be well reconstructed using the learned discrete codebook representation.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is implicitly modeled as producing abnormal human motion that deviates from normal behavior patterns learned exclusively from normal motion pose sequences. The method assumes no prior examples of anomalies during training, reflecting real-world rarity. The adversary cannot influence the pose keypoint detection input or distort pose extraction.

2. Data: VQ-VAD is trained primarily on the CMU Panoptic dataset for representation learning, which includes multi-view synchronized 2D and 3D keypoint sequences with 18 joints per frame. The authors segment into 48-frame windows, yielding 14,081 training and 2,672 validation segments (over 1.2 million pose frames). Anomaly detection evaluation is performed on four benchmarks (SHT, HR-SHT, HuVAD, NWPUC) with both training and test pose sequences, each dataset varying in size and camera views.

3. Architecture / Algorithm: The core of VQ-VAD is a vector-quantized autoencoder adapted from VQ-GAN that operates on flattened pose keypoint sequences (sequence length × channels). The encoder stacks 1D temporal convolutions producing continuous latent vectors. A vector quantizer then maps each latent vector to its nearest entry in a learned finite codebook of discrete motion embeddings. The decoder reconstructs the original pose sequence from these discrete tokens. A discriminator adversarially enforces motion realism on reconstructed sequences. The total loss combines MSE reconstruction loss, vector quantization loss ensuring encoder outputs adhere to codebook embeddings, and adversarial loss promoting smooth, realistic outputs.

4. Training Regime: Models are trained end-to-end with AdamW optimizer (learning rate 2.25e-4, beta1=0.5, beta2=0.9) on 2D pose data with 36 channels using NVIDIA RTX A6000 GPUs. The best performing compression factor is F4 (temporal downsampling by 4) and the codebook size set to 1024, balancing reconstruction quality and computational cost. Loss weights are set as reconstruction=1.0, vector quantization=1.0, adversarial=0.5. Specific epochs, batch size, and random seed strategies are not detailed, but training continues until convergence on validation reconstruction.

5. Evaluation Protocol: The learned model is evaluated first on reconstruction capability metrics: Mean Per-Joint Position Error (MPJPE), L1 reconstruction error, and temporal smoothness comparing ground truth and reconstructions. For anomaly detection, Area Under ROC Curve (AUC-ROC) is reported on multiple datasets. Evaluations cover three protocols: in-domain (train/test on same dataset), cross-domain (train on CMU Panoptic, test on other datasets without fine-tuning), and cross-dataset (train on one anomaly detection dataset, test on others). Comparisons against multiple recent pose-based VAD baselines like STG-NF, TSGAD are provided. Ablations study codebook size and temporal compression effects.

6. Reproducibility: The authors provide a public codebase at https://github.com/TeCSAR-UNCC/VQ-VAD, alongside pretrained weights and documented benchmarks for anomaly detection evaluation. Datasets like CMU Panoptic and SHT are publicly available, permitting reproduction of results. Some datasets used for anomaly detection (HuVAD, NWPUC) and proprietary under certain licenses.

Example end-to-end: A 48-frame 2D pose sequence (18 joints × 2D coordinate) is input, flattened along joints and coordinates into 1D temporal features, then encoded by 1D conv layers into continuous latent vectors. Each vector is quantized to nearest codebook motion token (from vocab size 1024). The decoder reconstructs the pose sequence from tokens. Reconstruction error (MSE) between input and output sequences produces anomaly score. Normal motions yield low error; abnormal motions yield high error due to poor quantization fit.

Technical innovations

• Adaptation of VQ-GAN vector quantization from images to sequential human pose keypoint data for anomaly detection.
• Learning a discrete motion vocabulary that compactly captures human motion semantics for pose-based VAD.
• Integration of adversarial training to improve motion realism in reconstructed pose sequences alongside vector quantization.
• Definition and use of cross-domain and cross-dataset evaluation protocols focusing on robustness and generalizability.

Datasets

• CMU Panoptic Dataset — ~1.3 million pose frames — public multi-view motion capture dataset used for representation learning
• SHT — 257,650 train frames, 37,845 test frames, 13 cameras — public anomaly detection benchmark
• HR-SHT — subset/variant of SHT with higher resolution or pose data — similar size to SHT
• NWPUC — 715,901 train frames, 284,228 test frames, 43 cameras — public semi-supervised anomaly benchmark
• HuVAD — 4,467,271 train frames, 729,404 test frames, 7 cameras — recent public human-centric anomaly detection dataset

Baselines vs proposed

• On HR-SHT in-domain: STG-NF AUC = 87.4% vs VQ-VAD AUC = 81.83%
• On HR-SHT cross-domain: VQ-VAD (trained on CMU Panoptic) AUC = 76.69% (no direct baseline)
• On HuVAD cross-dataset: STG-NF AUC = 54.32% vs VQ-VAD AUC = 59.66%
• On NWPUC cross-dataset: STG-NF AUC = 62.44% vs VQ-VAD AUC = 61.81%
• On SHT cross-domain: VQ-VAD AUC = 75.09% vs multiple baselines ranging 70-85%
• Increasing codebook size from 256 to 1024 reduces MPJPE from 0.10468 to 0.07222 and L1 error from 0.06625 to 0.04595

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.05069.

Fig 1: VQ-VAD architecture for unsupervised human anomaly detection based on pose sequences. The input pose

Limitations

• VQ-VAD relies on accurate pose keypoint sequences; noisy or erroneous pose detection may degrade performance.
• The approach has not been evaluated against active adversarial attempts to evade or manipulate motion tokens.
• Temporal compression and codebook sizes were selected via limited ablation; other configurations could be explored.
• Cross-domain evaluations generalize well but still show varying performance (some datasets as low as ~57% AUC).
• Complex multi-person interactions and occlusions remain challenging for pose-based anomaly detection.
• Detailed hyperparameter tuning details (e.g., epochs, batch sizes) are not fully disclosed, limiting exact reproducibility.

Open questions / follow-ons

• How robust is VQ-VAD to noisy or incomplete pose keypoints, especially in crowded or occluded scenes?
• Can adversarially crafted motion sequences bypass detection by exploiting the discrete motion vocabulary?
• What is the impact of incorporating multi-person interaction modeling explicitly in the vector-quantized framework?
• Can discrete motion token vocabularies be learned jointly with other modalities (e.g., appearance) for improved anomaly discrimination?

Why it matters for bot defense

For bot-defense or CAPTCHA practitioners, VQ-VAD’s approach to discretizing human motion into a compact codebook provides a structured, interpretable representation that could be leveraged to analyze human behavior more robustly. Unlike pixel-based approaches sensitive to appearance changes, pose-based discrete embeddings reduce noise and privacy risks, potentially enabling better detection of automated abnormal or synthetic motion patterns in surveillance or user interaction data. The demonstrated cross-domain and cross-dataset generalization indicates potential for deployment in varied real-world environments without per-site retraining. However, this method assumes high-quality keypoint data, which may be challenging in some CAPTCHA contexts. Additionally, the anomaly detection based on reconstruction error might inspire methods for identifying bot-like motion that do not conform to learned normal human motion vocabularies.

Cite

@article{arxiv2608_05069,
  title={ VQ-VAD: Vector-quantized Motion Representation Learning for Human-centric Video Anomaly Detection },
  author={ Narges Rashvand and Ghazal Alinezhad Noghre and Shanle Yao and Gabriel Maldonado and Hamed Tabkhi },
  journal={arXiv preprint arXiv:2608.05069},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.05069}
}

Read the full paper

• arXiv:2608.05069 (abstract)
• arXiv:2608.05069 (PDF)