TimeProVe: Propose, then Verify for Efficient Long Video Temporal Reasoning in Activities of Daily Living

Source: arXiv:2606.20561 · Published 2026-06-18 · By Arkaprava Sinha, Dominick Reilly, Siddharth Krishnan, Hieu Le, Srijan Das

TL;DR

The paper addresses the challenge of Long Video Question Answering (LVQA) where the goal is to answer natural language queries about hours-long untrimmed videos of Activities of Daily Living (ADL). Existing approaches suffer from either prohibitive computational cost by processing videos densely with large vision-language models (VLMs) or suffer accuracy losses when reasoning only over sparse captions that miss critical temporal and motion-based evidence. TimeProVe proposes a hybrid "propose-then-verify" framework that first generates query-conditioned candidate answer and evidence hypotheses using a lightweight Action-based Candidate Evidence (ACE) module. This module detects and temporally localizes atomic actions, converts them into query-relevant candidate windows through prompt-based lightweight LLM reasoning, and ranks them by temporal and semantic relevance. Only the top candidates are sent to an expensive cloud VLM for precise visual verification, avoiding full-video dense processing. To benchmark this temporally grounded and open-ended reasoning task, the authors introduce OpenTSUBench (OTB), a new LVQA dataset on real untrimmed ADL videos with temporally annotated evidence. Experiments show TimeProVe outperforms the strongest baselines by 7.3% on OTB while reducing costly VLM calls by 75% and inference cost by 93%. The framework also generalizes competitively to temporal grounding on Charades-STA, achieving state-of-the-art when combined with grounding VLMs. Thus the contribution is a novel hybrid design that combines lightweight temporal action detection and LLM reasoning to enable efficient and accurate long video temporal reasoning in ADL scenarios.

Key findings

• TimeProVe outperforms the strongest baseline on OpenTSUBench by 7.3% overall accuracy.
• The framework reduces the number of Vision-Language Model (VLM) calls by 75% compared to full-video inference.
• Inference cost is lowered by 93% due to targeted short-clip verification instead of dense processing.
• The Action-based Candidate Evidence (ACE) module run once over the full video generates a sparse temporal action timeline providing effective temporal structure.
• Query-conditioned proposal generation by a lightweight LLM improves accuracy by 3.6% over just detection alone (Table 2).
• Combined scoring and reranking of candidate windows yields the best accuracy at 42.7% (Table 2).
• On Charades-STA temporal grounding, TimeProVe achieves competitive [email protected] of 27.3% with just the Action Detector and state-of-the-art results when combined with Time-R1 grounding models (Table 4).
• Ablating temporal compatibility in scoring causes the largest drop in accuracy, confirming its importance (Fig 3).

Methodology — deep read

1. Threat Model & Assumptions: The adversary is not explicitly defined as this is not a security paper, but the system assumes that relevant evidence for answering queries lies in short, temporally localized actions or merged short windows, rather than diffuse long-range context. The adversary cannot corrupt the input video or control the query but the system operates efficiently under constraints of large video duration and limited VLM processing budget.

2. Data: The system is evaluated primarily on OpenTSUBench (OTB), a new benchmark derived from the Toyota Smarthome Untrimmed (TSU) dataset, containing 3,567 free-form QA pairs over 185 videos averaging 21 minutes each. Each question is paired with one or more supporting temporal intervals. Evaluation also includes Charades-STA for temporal grounding. The videos come with timestamped action annotations from TSU or Charades.

3. Architecture / Algorithm:

• The core lightweight module is the Action-based Candidate Evidence (ACE) Module, which has two parts: a) Action Detector: A single pass over video segments encoded by a frozen visual backbone (e.g. CLIP-L/14 or I3D) predicts action probabilities per temporal segment (using the MS-Temba model with 17M parameters). A thresholded output produces a sparse timeline of atomic action events with start/end times. b) Query-conditioned Proposal Generator: Takes the action timeline and query text, and uses an edge LLM (Gemma4-2B or Qwen2-7B) to generate candidate answer-evidence pairs. For each detected action, an atomic evidence window is created. The LLM can also group atomic windows into merged windows to capture temporal context for the query.
• These candidate windows are scored and reranked using a combined metric integrating temporal compatibility with query intent, semantic relevance to the query tokens, coverage of query tokens, and length penalty for window size. The scoring function efficiently ranks queries for verification.
• The Temporal Verifier: An expensive cloud-based VLM (VideoLLaMA3 or GPT-4o) receives only the top candidate clips from ACE to verify if visual evidence supports the candidate answer. Verification stops when one candidate is accepted or budget is exhausted.

4. Training Regime:

• The action detector MS-Temba is pretrained on Toyota Smarthome Untrimmed or Charades datasets.
• The edge LLMs run in zero-shot mode prompted with query and action timelines.
• No explicit temporal grounding training is required for TimeProVe.

5. Evaluation Protocol:

• Primary metric is overall question-answering accuracy on OpenTSUBENCH, which requires correct answer and temporal evidence.
• Breakdowns are provided across spatial, temporal, compositional, and long-horizon evidence categories.
• Efficiency metrics include number of VLM calls, video duration processed, and latency.
• Comparative baselines include supervised fine-tuned VLMs, agentic frameworks, caption-based and uniform sampling baselines.
• Ablations explore effects of individual ACE components and scoring metrics (Table 2, Fig 3).
• Additional testing on Charades-STA temporal grounding dataset using Recall@IoU thresholds.

6. Reproducibility:

• Code, detailed prompts, and training configurations are referenced but not confirmed fully open-source.
• The OpenTSUBENCH dataset is constructed from publicly available TSU annotations with additional processing.
• Edge LLM weights used are from known available models (Gemma4-2B, Qwen2-7B).

Concrete end-to-end example: Given a long ADL video and a query such as “What action did the person perform while drinking something?”, the system first runs the Action Detector once over the entire video to produce a timeline of atomic actions (e.g. drink at 58.2s-2m11s, read at 1m39s-2m32s). The Proposal Generator LLM uses this timeline plus the query to propose candidate answers—e.g. “read” with a merged evidence window containing the intervals for both 'drink' and 'read' actions. All candidates are scored for temporal and semantic relevance and compactness. The highest scoring candidate is sent as a short RGB clip from the original video along with the query and candidate answer to the cloud VLM verifier. The VLM confirms or rejects the evidence. If confirmed, the verified answer and supporting video clip and semantic evidence are returned as output, avoiding full-video dense processing.

Technical innovations

• Action-based Candidate Evidence (ACE) module that converts temporally localized atomic actions into query-conditioned candidate answers and variable-length supporting evidence windows via lightweight LLM reasoning.
• A combined scoring function integrating temporal compatibility, semantic relevance, content coverage, and length penalty to rank candidate evidence windows for efficient verification.
• A hybrid two-stage 'propose-then-verify' framework where a lightweight action detector processes the whole long video once and invokes large VLMs only for targeted short-clip visual verification.
• Introduction of OpenTSUBENCH (OTB), an open-ended LVQA benchmark emphasizing temporally grounded reasoning on real-world untrimmed ADL videos with timestamped evidence annotation.

Datasets

• OpenTSUBENCH (OTB) — 3,567 QA pairs over 185 videos, average duration 21 minutes — derived from Toyota Smarthome Untrimmed (TSU) dataset
• Charades-STA — approx. 6,000 videos with temporal grounding annotations — public dataset

Baselines vs proposed

• VideoLLaMA3 baseline on OTB: overall accuracy = 37.8% vs TIMEPROVE + VLMA3: 42.7%
• TIMEPROVE with Gemma4-2B ACE + VLMA3 verifier: 14.7% absolute improvement over VLMA3 baseline
• TIMEPROVE with Qwen2-7B ACE + VLMA3 verifier: 5.4% improvement over Gemma4-2B ACE
• TIMEPROVE + GPT-4o verifier: 45.1% overall accuracy on OTB vs GPT-4o standalone: 23.8%, a 21.3% gain
• Efficiency comparison: TIMEPROVE achieves 44.8% accuracy processing 123.6s duration with 8.3 VLM calls vs Caption-based baseline 24.7% accuracy with 16.8 VLM calls processing 1004.8s duration
• On Charades-STA temporal grounding, TIMEPROVE + Time-R1 achieves [email protected] of 62.0% vs baseline Time-R1 61.5%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.20561.

Fig 1: TIMEPROVE reduces long-video LVQA cost

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• Assumes answer-relevant evidence is localized to small or merged action windows; may not handle queries requiring diffuse or broad scene understanding over long intervals well.
• Final verification depends on the visual and reasoning ability of the chosen VLM—accuracy bounded by verifier capacity.
• Temporal boundaries from action detection are coarse; might not suffice for strict temporal grounding requiring fine alignment.
• OpenTSUBENCH focuses on ADL scenarios; generalization to other long video domains is untested.
• Relies on pretrained action detectors and LLMs; improvements may depend on advances in these pretrained components.
• Adaptive evidence aggregation and verifier calibration are future work; current window merging relies on LLM heuristics.

Open questions / follow-ons

• How can TimeProVe better aggregate and reason over diffuse or broadly distributed evidence beyond localized action windows?
• Can the verifier incorporate uncertainty estimation or active learning to adaptively select clips for improved efficiency and accuracy?
• How well does the framework generalize to domains with very different action taxonomies or less structured videos?
• What are the impacts of using stronger or different pretrained action detection and temporal grounding models within the ACE module?

Why it matters for bot defense

TimeProVe's core idea of a propose-then-verify hybrid framework offers a promising strategy for bot-defense or CAPTCHA systems that must efficiently analyze long video inputs or streaming video where only sparse evidence is relevant. The key insight is that exhaustive application of large, costly vision-language models over full temporal streams is computationally prohibitive. Instead, lightweight, local detectors combined with contextual LLM reasoning can generate candidate hypotheses that sharply reduce the workload for expensive verification models. For CAPTCHA or bot detection involving video challenges, this suggests architectures that preprocess with small local action recognizers or motion detectors and reserve complex verification for a minimal number of highly relevant clips or frames. Moreover, the temporal scoring and reranking techniques balance evidence relevance, temporal positioning, and verification cost, which could guide selection of minimal yet high-informative evidence snippets for human or automated verification. However, the approach depends on having suitable action detectors and clear temporal windows, so it may be less applicable for adversarial or heavily manipulated video content where evidence is diffuse or corrupted. Finally, TimeProVe’s emphasis on exposing both semantic and visual evidence provenance is beneficial for auditability and interpretability in security-sensitive applications. Overall, the paper presents foundational design patterns and analysis metrics that could be adapted to increase efficiency and reliability of CAPTCHA-like video challenges in bot-defense.

Cite

@article{arxiv2606_20561,
  title={ TimeProVe: Propose, then Verify for Efficient Long Video Temporal Reasoning in Activities of Daily Living },
  author={ Arkaprava Sinha and Dominick Reilly and Siddharth Krishnan and Hieu Le and Srijan Das },
  journal={arXiv preprint arXiv:2606.20561},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.20561}
}

Read the full paper

• arXiv:2606.20561 (abstract)
• arXiv:2606.20561 (PDF)