MG-RWKV: Multi-Grained Context-Aware RWKV for Temporal Forgery Localization

Source: arXiv:2607.00902 · Published 2026-07-01 · By Jingchen Ni, Cangjin Yu, Dan Jiang, Quan Zhang, Keyu Lv, Shannan Yan et al.

TL;DR

This paper addresses the problem of Temporal Forgery Localization (TFL), which requires precisely identifying manipulated segments within long untrimmed audio-visual sequences. Existing methods either rely on CNNs with limited local receptive fields or Transformers incurring quadratic complexity, making it challenging to efficiently capture both global context and local abrupt forgery cues. The authors propose MG-RWKV, a novel linear-complexity framework based on the RWKV recurrent architecture that leverages adaptive data-dependent state evolution for efficient full-sequence modeling at O(T) complexity.

MG-RWKV integrates three key innovations: (1) a Bidirectional RWKV architecture for capturing forward and backward temporal context simultaneously without quadratic overhead; (2) a Multi-Granularity Mixture of Experts (MG-MoE) module that dynamically routes inputs through experts specializing in different explicit temporal receptive fields to adapt to forgery durations and improve interpretability; and (3) a Cross-Granularity Consistency (CGC) loss that enforces feature alignment across adjacent pyramid scales, reducing false positives by resolving multi-scale inconsistencies. Extensive experiments on three diverse datasets—Lav-DF, TVIL, and Psynd—demonstrate that MG-RWKV achieves state-of-the-art temporal forgery localization performance with superior boundary precision and recall, while incurring substantially lower computational cost compared to Transformer-based alternatives.

Key findings

• MG-RWKV achieves 87.29% mAP on Lav-DF, surpassing RWKV-7 baseline by 4.86% and outperforming state-of-the-art Transformer-based UMMAFormer (94.81% [email protected] vs 92.96%) with lower complexity.
• On TVIL, MG-RWKV attains 71.31% [email protected], outperforming UMMAFormer by 8.88%, and achieving 92.24% AR@100, indicating superior boundary accuracy and recall.
• MG-RWKV surpasses UMMAFormer by 10.22% [email protected] on Psynd audio-only dataset, confirming multi-granular temporal modeling generalizes across modalities.
• Bidirectional RWKV improves mAP by 3.21% to 8.87% across datasets compared to unidirectional baseline, highlighting the importance of bidirectional context for precise boundary localization.
• MG-MoE contributes 0.95% to 1.46% mAP gains via adaptive scale selection among dilation-based experts corresponding to receptive fields of sizes 1, 2, and 4.
• The CGC loss yields largest improvements (up to 1.56% mAP, 5.44% [email protected]) by enforcing cross-scale consistency in authentic regions with boundary-aware weighting and progressive warmup.
• MG-RWKV maintains linear O(T) computational complexity in sequence length, showing better scaling of FLOPs and memory compared to quadratic Transformers (Fig. 1c,d).
• MG-RWKV inference latency on Lav-DF is 73.4ms with 56.2M params, comparing favorably against baselines with higher cost and lower accuracy.

Threat model

The adversary is assumed to produce temporally localized forgeries within long audio-visual sequences aiming to evade detection. The attacker can manipulate content segments but does not have unlimited generation freedom to remove all forensic traces. The detection model relies on temporal bidirectional context and multi-scale temporal patterns to identify forgery boundaries. The adversary cannot manipulate features arbitrarily at the backbone level or perform strong adversarial attacks against learned representations.

Methodology — deep read

1. Threat Model & Assumptions: The adversary aims to create temporally localized audiovisual forgeries within videos or audio sequences that blend into authentic content. The detection model assumes no prior knowledge of forgery locations or types but leverages temporal context around forgery boundaries to localize manipulated segments. The attacker does not have unlimited ability to evade feature-level cues learned by the detector.

2. Data: Experiments are conducted on three established datasets: Lav-DF (multi-modal, deepfake videos from VoxCeleb2), TVIL (video-only inpainting forgery dataset derived from YouTubeVOS), and Psynd (audio-only voice cloning from LibriTTS). Each dataset includes raw untrimmed sequences, dense temporal annotations of forged segments, and standard training/testing splits defined in prior work.

3. Architecture/Algorithm: MG-RWKV processes extracted audiovisual features (visual features from TSN, audio features from BYOL-A) fused and projected into a sequential embedding X ∈ RT×D.

• The backbone consists of stacked MG-RWKV blocks using a bidirectional RWKV-7 recurrent model adapted with independent parameter sets for forward and backward passes to capture past and future context with O(T) complexity.
• Each MG-RWKV block applies multi-scale dilated convolutions (kernel size 3 with dilations {1,2,4}) followed by bidirectional RWKV processing per dilation as specialized 'experts.'
• The Multi-Granularity Mixture of Experts (MG-MoE) module dynamically routes each temporal position's input to a sparse subset (top-2) of experts using a lightweight router network employing mean and max pooling across channels, followed by temperature-softmax and top-K gating to enable adaptive scale selection.
• The fused bidirectional expert outputs are combined via a linear projection.
• A Feature Pyramid Network (FPN) refines multi-scale outputs for classification and regression heads predicting dense forgery probabilities and temporal boundary offsets.
• The Cross-Granularity Consistency (CGC) loss enforces cosine similarity between adjacent FPN features exclusively in authentic regions identified by dilated ground-truth forgery masks, using a spatial boundary-aware weighting and a warmup schedule to avoid early collapse.

4. Training Regime: Training applies AdamW optimizer with initial learning rate 1e-4 and cosine annealing over 45 epochs for Lav-DF and TVIL, 30 epochs for Psynd. Batch size is not explicitly stated. Data augmentations include random cropping, label smoothing, and drop path. Losses combined include focal classification loss, DIoU boundary regression loss, an auxiliary reconstruction loss, and progressively weighted CGC loss.

5. Evaluation Protocol: Metrics use standard Average Precision (AP) and Average Recall (AR) at tIoU thresholds {0.5, 0.75, 0.95} and numbers of proposals {10, 20, 50, 100} consistent with prior TFL works. Soft-NMS extracts top-100 proposals from dense predictions. Multiple baselines including UMMAFormer, BA-TFD, and Mamba linear models are compared. Progressive ablation studies dissect BiDir, MG-MoE, and CGC contributions. Hyperparameter sensitivity tests are performed for CGC parameters.

6. Reproducibility: The paper does not mention code release or pretrained checkpoints. Datasets are publicly available. Detailed architectural settings, hyperparameters, and training schedules are fully described to enable replication. The RWKV-7 backbone is adopted from prior RWKV work with extensions, implying partial dependency on earlier open implementations.

Example Flow: Given an untrimmed video, TSN and BYOL-A extract visual and audio features, fused into a sequential input fed into L=9 MG-RWKV blocks. Each block performs multi-scale convolution, bidirectional RWKV passes (forward and backward), and MG-MoE dynamic routing selecting top-2 experts per frame. Outputs feed into an FPN, producing multi-scale feature pyramids. The CGC loss aligns these features during training via cross-scale cosine consistency in authentic regions, emphasizing sharpness at boundaries. Final dense forgery scores and boundary offset regressions are thresholded and filtered by Soft-NMS to produce temporal forgery proposals.

Technical innovations

• Bidirectional RWKV architecture extends linear-complexity RWKV recurrent model with independent forward and backward passes to capture bidirectional temporal context without quadratic self-attention costs.
• Multi-Granularity Mixture of Experts (MG-MoE) dynamically routes sequence positions to convolutionally dilated RWKV experts with explicitly interpretable temporal receptive fields, enabling adaptive granularity selection tuned to forgery durations.
• Cross-Granularity Consistency (CGC) loss enforces hierarchical scale-wise cosine similarity between adjacent FPN features only in authentic regions with boundary-aware weighting and progressive warmup scheduling, reducing false positives from multi-scale prediction inconsistency.
• Integration of gating-based multi-scale convolution preprocessing combined with sparse Top-K routing in MG-MoE provides a structured and interpretable multi-resolution temporal feature selection mechanism, uncommon in prior linear or Transformer-based TFL models.

Datasets

• Lav-DF — ∼VoxCeleb2-based, multimodal deepfake videos — publicly available
• TVIL — YouTubeVOS 2018 derived video-only forgery dataset — publicly available
• Psynd — LibriTTS-based audio-only voice-cloning forgery dataset — publicly available

Baselines vs proposed

• UMMAFormer Visual+Audio: [email protected] = 37.61% vs MG-RWKV Visual+Audio: 38.47%
• Mamba: mAP = 80.15% vs RWKV-7 baseline: 82.43% vs Full MG-RWKV: 87.29%
• UMMAFormer Visual (TVIL): [email protected] = 62.43% vs MG-RWKV Visual: 71.31%
• UMMAFormer Audio (Psynd): [email protected] = 79.87% vs MG-RWKV Audio: 90.09%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.00902.

Fig 1: Performance and computational efficiency comparison on TVIL dataset. (a)

Fig 2: Overview of the proposed MG-RWKV framework. (a) Overall pipeline with

Fig 3: Progressive component ablation on TVIL dataset. From top to bottom: Ground

Fig 4: Ablation study on MG-MoE configuration choices on the TVIL dataset. (a)

Fig 5: CGC hyperparameter sensitivity. (a) Consistency weight λ peaks at 0.01. (b)

Fig 6: MG-MoE dynamic granularity selection on TVIL. Coarse scales dominate in

Fig 7: Qualitative comparison on TVIL dataset. Top and bottom rows show two

Fig 8 (page 24).

Limitations

• The paper does not report robustness under adversarial manipulations attempting to evade RWKV-based TFL detectors.
• No experiments on out-of-distribution or highly diverse unseen forgery types beyond tested datasets, limiting generalization claims.
• The bidirectional RWKV extension requires training independent forward and backward parameters, increasing model size compared to unidirectional baseline.
• The CGC consistency loss introduces additional hyperparameters (ignore radius, boundary radius, warmup epochs) requiring careful tuning, which may limit plug-and-play deployment.
• While computationally efficient, MG-MoE increases inference time compared to baseline RWKV-7 by ~40ms; real-time deployment on constrained devices may be challenging.
• Lack of released code or pretrained weights at publication time limits immediate reproducibility and adoption.

Open questions / follow-ons

• How robust is MG-RWKV against adaptive adversarial attacks specifically crafted to fool linear recurrent TFL architectures?
• Can the MG-MoE dynamic routing and CGC consistency constraints generalize to other sequence tasks like temporal action detection or anomaly detection?
• How does MG-RWKV perform under real-world distribution shift scenarios such as compression artifacts or unseen video manipulation types?
• What are the trade-offs between more granular dilation scales or more complex expert routing strategies on interpretability and efficiency?

Why it matters for bot defense

Temporal forgery localization addresses a critical bot-defense challenge by enabling precise identification of manipulated segments within multimedia streams. For bot-defense engineers, MG-RWKV’s ability to efficiently process full-length sequences with linear complexity is highly valuable given the latency constraints in real-time verification pipelines. The bidirectional RWKV and MG-MoE modules balance global context understanding with fine-grained local anomaly sensitivity, essential for subtle forgery detection.

Furthermore, the cross-granularity consistency constraint reduces false positives common in multi-scale approaches, improving robustness against noisy or authentic regions. Applying MG-RWKV concepts could inform CAPTCHA and bot-detection systems aiming to verify media integrity over time, especially as AI-generated content grows more sophisticated. However, adapting MG-RWKV requires access to audiovisual feature extractors and training on relevant forgery datasets. The method’s linear computational cost offers a promising avenue to scale forgery detection in large-scale automated bot-defense frameworks without prohibitive resource consumption.

Cite

@article{arxiv2607_00902,
  title={ MG-RWKV: Multi-Grained Context-Aware RWKV for Temporal Forgery Localization },
  author={ Jingchen Ni and Cangjin Yu and Dan Jiang and Quan Zhang and Keyu Lv and Shannan Yan and Linyue Pan and Ke Zhang and Chun Yuan },
  journal={arXiv preprint arXiv:2607.00902},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.00902}
}

Read the full paper

• arXiv:2607.00902 (abstract)
• arXiv:2607.00902 (PDF)