LongVQUBench: Benchmarking Long-Term Video Quality Understanding of Vision-Language Models

Source: arXiv:2607.01086 · Published 2026-07-01 · By Arpita Nema, Hanwei Zhu, Xi Zhang, Weisi Lin

TL;DR

This paper addresses the underexplored problem of long-term video quality understanding (LVQU) by large vision-language models (LVLMs). Existing benchmarks focus on short clips with isolated distortions, ignoring temporal continuity, cumulative quality degradation, and reasoning complexity present in long videos. To fill this gap, the authors introduce LongVQUBench, a comprehensive benchmark comprising over 1,200 diverse videos spanning minutes to almost two hours, sourced from movies, documentaries, surveillance, egocentric, and animated content. Accompanying the videos are 1,500 question-answer pairs designed to evaluate perceptual reasoning at three hierarchical levels: local event quality understanding (LQU), cross-event quality reasoning (CQR), and global quality understanding (GQU). These progressively test models’ capabilities from localized distortion detection to holistic perceptual evaluation across extended durations. Additionally, a novel needle distortion question-answering (NDQA) paradigm embeds fine-grained spatial and temporal artifacts sparsely to probe fine perceptual sensitivity. Evaluation of 14 state-of-the-art LVLMs reveals significant performance degradation as video length and reasoning complexity increase, exposing current models’ limitations in long-range temporal integration and perceptual attribution. This benchmark lays a foundational framework for systematic, hierarchical evaluation of LVLMs’ long-term video quality understanding, beyond existing short-clip or semantic-centric tasks.

Key findings

• LongVQUBench contains 1,200+ videos with durations ranging from a few minutes up to 2 hours (average length 742.2 seconds), substantially longer than prior benchmarks.
• The benchmark includes 1,500 multiple-choice and open-ended QA pairs targeting hierarchical perceptual reasoning at local event (LQU), cross-event (CQR), and global video (GQU) levels.
• 14 evaluated LVLMs show consistent performance decline as reasoning level deepens: for example, GPT-5 scores 81.2 accuracy on CQR but only 65.8 on GQU, indicating difficulty with holistic long-term quality integration.
• Increasing number of uniformly sampled frames (up to 256 frames at 1 FPS) yields limited performance improvement for most LVLMs, showing bottlenecks in temporal modeling rather than input coverage.
• Agentic LVLMs with adaptive frame selection, such as DeepVideoDiscovery, achieve better scores (71.7 total accuracy) than uniform sampling baselines, but effectiveness varies substantially by approach.
• The NDQA paradigm with sparse, controlled spatial and temporal distortions enables probing fine-grained perceptual sensitivity beyond coarse semantic cues.
• Most LVLMs excel in detecting and reasoning about localized distortions (LQU) and multi-event comparisons (CQR) but struggle with holistic judgments requiring temporal aggregation (GQU).
• Validation split shows that video-specialized open-source models like VQA2 peak at relatively low frame counts (8 frames) while proprietary models benefit moderately from more frames.

Threat model

The benchmark implicitly evaluates the limitations of LVLMs as adversaries in perceiving and reasoning about video quality degradations over extended time horizons. The adversary is effectively the model’s capability constraints—incapacity to integrate long-range temporal cues or finely detect subtle spatiotemporal artifacts—rather than an explicit malicious attacker. The evaluation assumes availability of sampled frames and prompt instructions but no iterative feedback or external knowledge beyond the video frames provided.

Methodology — deep read

The paper’s methodology centers on constructing a large-scale benchmark, LongVQUBench, to evaluate long-term video quality understanding by LVLMs through carefully designed data, evaluation levels, and question-answer pairs.

1. Threat Model & Assumptions: The goal is to test LVLMs’ ability to perceive and reason about perceptual quality degradations over extended temporal horizons. The adversary, in a broad sense, could be model limitations in temporal integration and perceptual reasoning. There is no explicit attacker; rather, the benchmark probes intrinsic LVLM weaknesses in long-term perceptual fidelity assessment.

2. Data: 1,200+ long-form videos collected from publicly available datasets (LongVideoBench, MLVU, LongVideoReason) spanning diverse content types (movies, documentaries, surveillance, egocentric, animated). Video durations range from a few minutes to up to two hours (average ~12 minutes). Videos are annotated with raw perceptual quality labels (high, medium, low). Controlled spatial and temporal distortions (14 spatial and 4 temporal types at 3 intensity levels each) are selectively applied to high-quality videos to generate ground-truth distortion-aware test samples.

3. Architecture / Algorithm: The benchmark evaluates 14 state-of-the-art LVLMs with diverse architectures including proprietary and open-source models, some video-specialized (e.g., VQA2). Models receive a uniform sampling of video frames (up to 1 FPS) along with textual prompts instructing them to analyze perceptual quality. The models output answers in a strict format to multiple-choice or open-ended questions without iterative feedback.

4. Training Regime: Models are tested under zero-shot settings—no fine-tuning on LongVQUBench data. The LVLMs utilize their pretrained weights and reasoning capabilities. Frame sampling rates and number of frames are hyperparameters tuned via a stratified validation split.

5. Evaluation Protocol: Questions are grouped into three hierarchical levels—Local Event Quality Understanding (LQU), Cross-Event Quality Reasoning (CQR), and Global Quality Understanding (GQU). For each level, multiple question types probe detection, localization, classification of distortions, their severity, cumulative effects, temporal relations, and holistic quality judgments. Both multiple-choice and open-ended questions are used; open-ended answers are scored via GPT-based relevance and completeness metrics. A stratified 40% validation / 60% test split ensures proportional representation of all question types.

6. Reproducibility: The authors publicly release the benchmark dataset and evaluation code via the project page. Model weights remain closed for proprietary systems, but open-source model implementations are referenced for replicability. Supplementary materials detail distortion configurations and annotation protocols.

One example end-to-end test: A high-quality documentary video (~20 minutes) is segmented into 15-second clips, then spatial blur distortion is applied to four random clips. The model receives 256 sampled frames at 1 FPS, along with prompts specifying the task of identifying localized distortions (LQU) and reasoning about their cumulative effect (CQR). The model must answer multiple-choice questions about distortion presence, severity, and temporal trends, with performance aggregated to assess its ability to perceive and reason about quality degradation over the long temporal context.

Technical innovations

• Introduction of a hierarchical evaluation framework (LQU, CQR, GQU) to systematically assess LVLMs’ video quality reasoning from local distortions to holistic temporal aggregation.
• Design and integration of a needle distortion question-answering (NDQA) paradigm that sparsely inserts controlled spatial and temporal distortions to probe fine-grained perceptual sensitivity within long videos.
• Compilation of a large-scale, diverse dataset of long-duration videos (up to 2 hours) from multiple domains with extensive distortion annotations to enable realistic long-term video quality understanding evaluation.
• Comprehensive zero-shot evaluation protocol for 14 state-of-the-art LVLMs using uniform and adaptive frame sampling strategies to investigate temporal coverage and reasoning capacity limits.

Datasets

• LongVQUBench — 1,200+ videos, 1,500 QA pairs — Aggregated from LongVideoBench, MLVU, LongVideoReason and curated/distorted by authors

Baselines vs proposed

• GPT-5: Total accuracy = 74.1 (256 frames) vs lower performance at fewer frames (70.5 at 8 frames)
• Gemini-3: Total accuracy = 68.9 (128 frames) vs 68.1 at 8 frames
• VQA2 (video-specialized open-source): Peaks at 59.4 total accuracy with 8 frames, declines with more frames
• DeepVideoDiscovery (agentic LVLM): Achieves 71.7 total accuracy with strong LQU=69.2 and CQR=82.7, outperforming simpler agentic models
• Across all LVLMs, LQU and CQR consistently score higher than GQU, e.g., GPT-5 CQR=81.2 vs GQU=65.8, indicating difficulty in global quality reasoning
• Adaptive frame sampling improves performance but results vary widely among agentic LVLMs

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.01086.

Fig 1: Long-term duration videos from LongVideoBench [57], MLVU [77], and

Fig 2 (page 25).

Fig 3 (page 31).

Fig 4 (page 31).

Fig 5 (page 31).

Fig 6 (page 31).

Fig 7 (page 31).

Fig 8 (page 31).

Limitations

• Evaluation limited to zero-shot settings without fine-tuning or adaptation to the LongVQUBench domain, possibly underestimating model capabilities.
• Frame sampling capped at 1 FPS, potentially missing finer-grained temporal nuances or distortions occurring at higher frequencies.
• Open-ended question evaluation relies on GPT-based scoring which, while advanced, may introduce bias or lack full rigor compared to human judgment.
• Proprietary model details and weights are not released, limiting exact reproducibility of top-performing systems' results.
• Distortion types and intensities, though diverse and controlled, may not cover all real-world video degradation phenomena.

Open questions / follow-ons

• How can LVLMs be architecturally improved to better integrate long-range temporal dependencies for perceptual quality reasoning?
• What are the effects of fine-tuning LVLMs specifically on long-form video quality datasets like LongVQUBench?
• Can adaptive or learned frame sampling strategies surpass uniform sampling in balancing efficiency and fidelity for long video quality understanding?
• How can open-ended reasoning about video distortions be more rigorously and human-aligned evaluated beyond GPT-based scoring?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, LongVQUBench’s advanced framework for evaluating long-term video quality perception by LVLMs highlights both the progress and limitations of current large vision-language systems in reasoning about subtle, temporally extended perceptual degradations. Applying similar hierarchical and fine-grained evaluation principles can help design CAPTCHAs and bot-detection challenges that exploit temporal coherence and cumulative quality effects — aspects difficult for automated attacks to model robustly over long video sequences. Incorporating sparse but targeted distortions, as in the NDQA paradigm, can enhance the challenge granularity, forcing adversaries to demonstrate deep perceptual reasoning rather than superficial scene understanding. Overall, this benchmark sets a benchmark for structured, explainable evaluation of vision-language models’ perceptual sensitivity, which is directly applicable for designing next-generation human verification tasks grounded in long-term visual processing.

Cite

@article{arxiv2607_01086,
  title={ LongVQUBench: Benchmarking Long-Term Video Quality Understanding of Vision-Language Models },
  author={ Arpita Nema and Hanwei Zhu and Xi Zhang and Weisi Lin },
  journal={arXiv preprint arXiv:2607.01086},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.01086}
}

Read the full paper

• arXiv:2607.01086 (abstract)
• arXiv:2607.01086 (PDF)