Deepfake News Detection: A Multimodal Framework Integrating LipNet, DeepSpeech and ResNET for Enhanced Audio-Visual Analysis

Source: arXiv:2607.20579 · Published 2026-07-22 · By Ameena Khan, Muhammad Ahsan Aziz, Muhammad Junaid Asif, Naeem Akhter, Rana Fayyaz Ahmad

TL;DR

This paper addresses the growing challenge of detecting deepfake news videos that manipulate both facial visuals and speech audio to deceive audiences. The authors propose a novel multimodal detection framework that integrates semantic features extracted from aligned lip movements (LipNet), speech content (DeepSpeech2), and facial appearance (BlazeFace + ResNet18). By concatenating these complementary audio-visual embeddings and classifying them via an ensemble of learning algorithms, the approach aims to improve robustness and detection accuracy over unimodal or handcrafted feature methods. Extensive evaluations on the FakeAVCeleb dataset—a challenging benchmark containing both audio and video manipulations—demonstrate that the proposed system achieves 94% accuracy with an augmented audio feature set, outperforming a recent state-of-the-art multimodal baseline. Interestingly, adding visual features to the audio modalities does not improve accuracy due to the near-indistinguishability of synthetic faces generated by state-of-the-art deepfake methods, emphasizing the importance of lip-sync and audio consistency cues. The paper also reports that the model performs poorly on the DFDC dataset, attributed to its lack of fine-grained modality labels, underscoring the need for suitable datasets for joint audio-visual deepfake news detection. The work balances detection accuracy with computational efficiency by selectively retaining only the most informative sub-networks and employing a lightweight Random Forest classifier.

Key findings

• The multimodal framework integrating LipNet (lip movements), DeepSpeech2 (speech), BlazeFace and ResNet18 (faces) achieves 94.36% accuracy on the FakeAVCeleb dataset's augmented audio features modality using Random Forest classifier.
• Data augmentation (sliding one-second segments) improves Random Forest audio-only accuracy from 86.15% to 94.36%, boosting F1-score from 0.8712 to 0.9424.
• Combining visual features (ResNet18) with audio features after augmentation yields slightly lower accuracy (93.34%) than audio-only (94.36%) due to visually indistinguishable synthetic faces.
• Random Forest classifier consistently outperforms MLP, LSTM, SVM, AdaBoost, Decision Trees, Naive Bayes, and K-Nearest Neighbor on the concatenated multimodal feature set.
• The DFDC dataset results in only ~52% accuracy due to missing modality-specific labels and unsynchronized audio, showing dataset limitations for audio-visual detection.
• LipNet spatial-temporal lip-motion features and DeepSpeech2 speech embeddings capture complementary semantic information critical for high classification performance.
• The selective retention of most informative sub-networks based on covariance analysis balances high accuracy with computational efficiency.
• Distance-based classifiers like K-Nearest Neighbor perform poorly (<52% accuracy), indicating the high-dimensional multimodal feature space is non-linear and better handled by ensemble methods.

Threat model

The adversary is a malicious entity capable of generating sophisticated deepfake news videos that manipulate both facial features and speech audio to convincingly impersonate real individuals for deception. The attacker can produce highly realistic synthetic faces and lip-synced audio, but is assumed not to have insider knowledge of the detection model’s internal parameters or training data. The detector assumes no direct adversarial attacks on the classifier itself but must recognize inconsistencies in audio-visual semantic coherence to flag manipulated content.

Methodology — deep read

The paper approaches deepfake news detection by jointly exploiting audio and visual modalities through transfer-learning and ensemble classification.

1. Threat Model & Assumptions: The adversary produces manipulated multimedia—facial videos and audio speech—to maliciously deceive viewers. The detector assumes the attacker may generate high-quality synthetic faces and lip-sync audio manipulations but does not specify adversarial knowledge beyond this. The goal is to distinguish authentic videos from fakes based on intrinsic audio-visual inconsistencies.

2. Data: The main dataset is FakeAVCeleb consisting of 500 real celebrity interview videos and roughly 19,500 fake videos generated via lip-sync deepfake methods. The dataset covers four categories of audio/video authenticity and features balanced ethnic and gender representation. The Deepfake Detection Challenge (DFDC) dataset is also used but found less suitable due to missing audio authenticity labels. Data augmentation is applied via a sliding window over 1-second video/audio segments, expanding the dataset to ~150,000 samples and helping address class imbalance.

3. Architecture / Algorithm: The framework extracts semantic features separately from three modalities:

   • Lip movements: LipNet processes sequential mouth-region video frames through spatio-temporal convolutions and bidirectional GRU layers to learn lip-motion embeddings.
   • Speech audio: DeepSpeech2 converts 1-second audio into Mel spectrograms, processed by CNN and recurrent layers with CTC loss, extracting speech features.
   • Facial imagery: Faces are detected and landmarked by BlazeFace and then resized and passed to ResNet18 for feature embeddings.

The resulting feature vectors are concatenated into a single multimodal representation. A covariance analysis discards less informative sub-networks to optimize for accuracy and computational cost.

4. Training Regime: Training uses transfer learning by fine-tuning existing pretrained models with cross-entropy loss and gradient descent optimizer at 0.01 learning rate for 100 steps. Dataset splitting is 70% train, 15% validation, 15% test. Models are evaluated proper cross-validation to avoid overfitting.

5. Classification Models: An ensemble of classifiers—Random Forest, MLP, LSTM, AdaBoost, Decision Trees, SVM (linear and polynomial), Naive Bayes, and K-Nearest Neighbor—is trained and compared on the concatenated features. Videos are treated as single time-step sequences, simplifying the LSTM input.

6. Evaluation: Metrics include accuracy, precision, recall, and F1-score focused on the fake class as positive. Experiments analyze unimodal audio, unimodal visual, and combined audio-visual setups with and without augmentation. Detailed tables show classifier performances, highlighting Random Forest as the best method.

7. Reproducibility: The paper partially details implementation (model architectures, hyperparameters) but does not mention the release of code or pretrained weights. The datasets used are publicly available except where noted.

End-to-end on FakeAVCeleb: Video frames of the speaker’s face are processed through BlazeFace and ResNet18 for visual embeddings. The cropped mouth region frames pass through LipNet to produce lip-motion features. The associated audio waveform is converted to Mel spectrogram segments and fed to DeepSpeech2 for speech-based features. The resulting feature vectors are concatenated and given to a Random Forest classifier that outputs a binary real/fake label. Data augmentation by sliding 1-second segments helps improve classification generalization on this highly imbalanced dataset.

Technical innovations

• Integration of three state-of-the-art pretrained models—LipNet (lip reading), DeepSpeech2 (speech-to-text), and ResNet18 (face features) via BlazeFace detection—into a unified multimodal deepfake detection framework.
• Semantic-based feature analysis that independently assesses each sub-network and retains only the most informative feature vectors based on covariance, optimizing accuracy and computational efficiency.
• Use of sliding window 1-second segmentation data augmentation to address severe class imbalance and improve model generalization over realistic audio-visual conditions.
• Comprehensive evaluation comparing an ensemble of machine learning and deep learning classifiers on the concatenated multimodal features, demonstrating Random Forest dominance in high-dimensional, non-linear feature space.

Datasets

• FakeAVCeleb — ~20,000 videos (500 real + 19,500 fake) — public, balanced ethnicity and gender, includes jointly manipulated audio-video samples with lip-sync forgeries
• DFDC (DeepFake Detection Challenge) — 128,154 videos — public, contains face-swap videos with unsynchronized audio and lacks modality-specific labels

Baselines vs proposed

• State-of-the-art ensemble multimodal baseline [25]: Accuracy = 89%
• Proposed Random Forest (audio features before augmentation): Accuracy = 86.15%
• Proposed Random Forest (audio features after augmentation): Accuracy = 94.36% vs baseline 89%
• Proposed Random Forest (audio + visual features after augmentation): Accuracy = 93.34% (slightly below audio-only)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.20579.

Fig 1: Overview of the deepfake generation process based on an autoencoder

Fig 2: Overview of the proposed multimodal feature extraction

Fig 3: Classification stage of the proposed framework. The concatenated feature vector is fed to an ensemble of classifiers including Random Forest, MLP,

Fig 4: Architecture of the LipNet feature extraction module. Sequential

Fig 5: DeepSpeech2 model: invariant convolution layers, batch normalisation

Fig 6: BlazeFace model output: a bounding box and six facial landmarks

Fig 7: Step-wise augmentation using a sliding sample window over one-

Fig 8: Accuracy of the classification models for audio features before

Limitations

• The DFDC dataset used for evaluation has incomplete modality-specific labels (audio authenticity not labeled), limiting its suitability and resulting in poor 52% accuracy.
• Visual features did not improve detection accuracy due to the high quality and realism of latest deepfake face synthesis, reducing the contribution of facial appearance cues.
• The methodology relies heavily on the quality and diversity of the FakeAVCeleb dataset; generalization to other types of deepfakes or broader video contexts is uncertain.
• The paper lacks explicit adversarial robustness evaluations against attackers aware of the detector.
• No details provided on computational cost or real-time feasibility beyond broad claims of efficiency; performance on large-scale streaming news data remains untested.
• Code and pretrained models are not indicated as publicly released, potentially limiting reproducibility.

Open questions / follow-ons

• How can multimodal deepfake detection frameworks be made robust against adaptive adversarial attacks that specifically target semantic lip-sync or audio features?
• Can the selective sub-network feature retention strategy be further optimized or learned dynamically during end-to-end training for improved accuracy-efficiency tradeoffs?
• How does the framework generalize to other deepfake manipulation types beyond lip-synced videos, such as entirely synthetic face generation without original audio?
• What are effective methods to build or augment datasets with reliably labeled per-modality ground truth to improve joint audio-visual detection?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this research highlights key insights about detecting synthetic manipulations in multimedia that go beyond traditional image-only analysis. The demonstrated effectiveness of audio-centric features, particularly lip-motion coherence with speech, suggests that incorporating multimodal semantic cues can strengthen defenses against AI-driven fake media bots that manipulate both video and audio layers. The finding that facial appearance features are increasingly unreliable due to advances in synthesis implies that CAPTCHA approaches relying only on visual artifacts may be insufficient in adversarial contexts involving deepfakes. The efficient fusion of pretrained modality-specific models combined with lightweight classifiers like Random Forest indicates practical pathways to deploying near real-time pipelines for verifying multimedia authenticity in online platforms. However, practitioners should note the limitations regarding dataset specificity and the need for modality-aware ground truth data, which are critical for training robust detection models tailored to emerging manipulation techniques. Overall, this work underscores the importance of multimodal analysis and augmentation techniques in evolving bot-defense strategies targeting sophisticated synthetic media.

Cite

@article{arxiv2607_20579,
  title={ Deepfake News Detection: A Multimodal Framework Integrating LipNet, DeepSpeech and ResNET for Enhanced Audio-Visual Analysis },
  author={ Ameena Khan and Muhammad Ahsan Aziz and Muhammad Junaid Asif and Naeem Akhter and Rana Fayyaz Ahmad },
  journal={arXiv preprint arXiv:2607.20579},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.20579}
}

Read the full paper

• arXiv:2607.20579 (abstract)
• arXiv:2607.20579 (PDF)