VidMap: Exploiting Temporal Structure for Video-Based Structure-from-Motion

Source: arXiv:2607.27194 · Published 2026-07-29 · By Zador Pataki, Paul-Edouard Sarlin, Marc Pollefeys

TL;DR

This paper addresses the challenge of reconstructing metric camera poses and calibration from unconstrained, long monocular videos, which current approaches struggle with due to various limitations. Classical Simultaneous Localization and Mapping (SLAM) methods operate incrementally and causally, making them vulnerable to initialization sensitivity, transient tracking failures, and inability to recover from drift, often requiring known intrinsics. Meanwhile, Structure-from-Motion (SfM) methods, although globally optimizing over all images, typically ignore temporal ordering and thus suffer from visual aliasing and poor robustness under extreme motion and visual symmetries.

To bridge the gap, the authors propose VidMap, a system that combines SLAM's strong sequential constraints with SfM's global, offline optimization. VidMap exploits temporal ordering as a first-class input to reliably distinguish between sequential observations and loop closure edges, improving robustness against visual symmetries and preventing drift. It also integrates learned wide-baseline dense image matching and monocular depth priors to regularize scale and degenerate motions. Extensive evaluations across diverse, challenging datasets including LaMAR, CroCoDL, ETH3D-SLAM, and EuRoC demonstrate that VidMap consistently outperforms state-of-the-art SLAM, classical and learned SfM baselines in both calibrated and uncalibrated settings, sometimes by large margins in long, complex sequences. This confirms the benefits of combining causal tracking concepts with global batch optimization and metric priors for robust video-based camera pose estimation.

Key findings

• VidMap improves translation error AUC by up to 10% absolute over best prior methods on LaMAR dataset in both calibrated and uncalibrated conditions (e.g., 93.2% vs 86.6% at 10m window AUC).
• On CroCoDL, VidMap achieves 94.7% and 91.5% AUC at 10m for phone and robot sequences respectively, outperforming closest baselines by 10-40%.
• VidMap estimates focal lengths from video alone with accuracy matching ground-truth calibrated performance.
• Provenance-aware track construction (preserving sequential vs loop closure labels) enables robust outlier rejection and prevents drift caused by incorrect loop closures, as visualized in Fig. 5.
• Monocular depth priors regularize scale and degenerate motions, mitigating scale drift especially in forward motion and low baseline sequences.
• On shorter and well-conditioned datasets (ETH3D-SLAM, EuRoC), VidMap outperforms DPV-SLAM by ~7-10% AUC while maintaining comparable performance on challenging fisheye and motion-blurred data.
• Multi-flow drift correction by chaining dense matches from multiple anchors reduces sequential propagation errors while rejecting inconsistent cross-keyframe matches.

Threat model

Adversary is any unconstrained monocular video input exhibiting challenging motion patterns, unknown calibration, visual symmetries, and textureless regions that can cause tracking failures or pose estimation drift. The system assumes no adversarial manipulation but aims to robustly handle natural failure modes where incremental or global SfM approaches typically break down.

Methodology — deep read

1. Threat model & assumptions: The adversary is any video sequence with unknown or known camera calibration, presenting challenges such as degenerate motions (forward motion, low parallax), visual symmetries, textureless surfaces, and extreme camera motions. The system assumes monocular video without controlled conditions and aims to recover metric poses and intrinsics robustly.

2. Data: The system uses monocular videos from various large-scale public datasets (LaMAR, CroCoDL, ETH3D-SLAM, EuRoC) featuring ground truth poses and calibration for evaluation. LaMAR and CroCoDL contain long, challenging, unconstrained sequences including phones and robots in complex indoor environments.

3. Architecture / algorithm: VidMap consists of two main stages: (a) video-aware sparse track extraction, and (b) global pose and structure optimization.

• It begins with dense pixel-wise matching via a learned dense matcher producing flow fields, uncertainty maps, and localization covariances between adjacent frames and loop closure candidates.
• A keyframe selection step adaptively chooses frames driven by tracked image motion.
• Sparse feature tracks are formed by chaining flows sequentially with multi-anchor drift correction using direct dense matches from multiple previous keyframes, selecting the most consistent prediction.
• Loop closures are detected via image retrieval and dense matching, but loop closure correspondences are kept separate in track provenance from sequential edges to distinguish in the optimization.
• Monocular depth maps with uncertainties are estimated per keyframe to provide metric depth priors.
• Camera intrinsics (primarily focal length) are estimated per keyframe via a learned monocular method (GeoCalib), then jointly refined by view-graph calibration and bundle adjustment.
• Global mapping uses robust rotation averaging with provenance-dependent losses (Huber for sequential edges trusted more, Cauchy for loop closures downweighting outliers).
• Global positioning jointly optimizes camera centers, 3D points, and per-image depth scale factors integrating monocular depth priors to regularize scale drift and degenerate motions.
• Final bundle adjustment refines poses, structure, intrinsics, and depth scales with robust reprojection and depth residuals.

4. Training regime: The learned components (dense matcher, monocular depth and calibration networks) are pretrained externally on large-scale datasets; VidMap itself is a system integrating these models without end-to-end training. The global optimization uses iterative non-linear least squares with graduated non-convexity for robustness.

5. Evaluation protocol: Accuracy is measured as area under the recall curve (AUC) of translation error over local windows (LaMAR, CroCoDL) or global alignment (ETH3D-SLAM, EuRoC). Baselines include classical and learned SLAM, SfM and end-to-end models, tested with fixed hyperparameters across datasets. Ablations show improvements due to video-aware extraction, provenance-aware losses, and depth priors.

6. Reproducibility: Code and models are publicly released (https://github.com/cvg/vidmap). The datasets are publicly available or standard benchmarks. Full details on hyperparameters, evaluation protocols, and ablations are provided, enabling reproducibility.

Example end-to-end: A long, uncalibrated indoor video is first segmented into keyframes via adaptive motion detection. Dense flows and monocular depth are computed per keyframe. Sparse feature tracks are formed by chaining flows with covariance-based consistency checks reducing drift. Loop closures are detected via image retrieval and matched densely, but LC matches are tagged separately. Focal length is estimated with GeoCalib and refined via view-graph calibration. Rotation averaging runs robustly with separate losses for sequential and LC edges to reject outliers. Global positioning optimizes camera centers, 3D points, and depth scales using monocular depth priors. Finally, bundle adjustment jointly optimizes all parameters for the best fit. Pose accuracy is then evaluated via local windowed translation error AUC.

Technical innovations

• Introducing provenance-aware track construction that separately maintains sequential and loop closure correspondences to prevent erroneous loop closure associations from corrupting global SfM optimization.
• Incorporating monocular depth priors with per-image scale factors into global positioning to regularize scale drift and degenerate video motions within an SfM optimization framework.
• Adapting dense learned pixel-level flow chaining from multiple prior keyframes for multi-flow drift correction, selecting the most consistent track prediction by covariance minimization.
• Combining the temporal ordering constraints of SLAM with the global offline optimization approach of SfM in a unified system exploiting the full video temporal structure while enabling loop closure upfront.

Datasets

• LaMAR — 50 phone videos in large indoor scenes with GT poses and calibrations — public
• CroCoDL — 88 sequences from phones and robots in disaster buildings with GT poses — public
• ETH3D-SLAM — 55 indoor/outdoor sequences with hand-held camera and GT poses — public
• EuRoC — 11 drone sequences with grayscale stereo fisheye cameras and GT poses — public

Baselines vs proposed

• On LaMAR uncalibrated, VidMap W-AUC@10m = 91.9% vs best prior DA3-Long = 86.7%
• On CroCoDL phones uncalibrated, VidMap W-AUC@10m = 94.0% vs DA3-Long 86.2%
• On CroCoDL robots uncalibrated, VidMap W-AUC@10m = 91.4% vs Lingbot-Map 60.9%
• On ETH3D-SLAM uncalibrated, VidMap AUC@5cm = 51.4% vs DPV-SLAM 38.7%
• On ETH3D-SLAM calibrated, VidMap AUC@5cm = 69.4% vs DROID-W 63.8%
• On EuRoC calibrated, VidMap AUC@5cm = 23.0% vs DROID-W 20.9%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.27194.

Fig 1: VidMap reconstructs long, unconstrained videos with complex mo-

Fig 2 (page 2).

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 2).

Limitations

• Evaluation focuses primarily on indoor, hand-held or robot and drone videos; performance in highly dynamic outdoor scenes or low-light conditions not fully explored.
• Loop closures rely on image retrieval and dense matching but could fail in very repetitive or textureless environments with extreme viewpoint changes.
• Monocular depth priors come from pretrained models and may degrade in out-of-distribution domains, though VidMap generalizes well on tested datasets.
• Runtime and scalability details for very long sequences are not deeply analyzed; offline global optimization may not be suitable for real-time or resource-constrained platforms.
• No explicit adversarial or robustness evaluation against maliciously crafted videos or spoofing attacks on loop closure.

Open questions / follow-ons

• How can the integration of learned priors and global geometric optimization be further improved for scale and loop closure in dynamic or outdoor scenes with moving objects?
• Can the approach be extended for or integrated with multi-camera or stereo video to improve robustness and scale?
• What are the tradeoffs between offline global optimization and streaming or real-time capable approaches for very long video sequences?
• How might adversarial conditions or intentional loop closure spoofing attacks be detected and mitigated within this framework?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, VidMap's approach to exploiting temporal and sequential structure to disambiguate perceptual aliasing and improve robustness under challenging visual symmetries is particularly relevant. Many bot defense scenarios rely on detecting or verifying camera movement or environment consistency from possibly adversarial video inputs. VidMap's provenance-aware global optimization and metric prior integration can inspire improved analysis techniques to detect replay attacks or synthetic video manipulations that might confuse standard frame-agnostic SfM or tracking models.

Furthermore, the dense matching combined with robust outlier rejection mechanisms offers a pathway to more reliable verification of user-generated video streams in CAPTCHA challenges or behavioral biometric systems. Although VidMap targets camera pose estimation rather than direct bot classification, the underlying ideas on integrating temporal ordering and learned monocular priors into pose and structure estimation pipelines provide valuable architectural insights to bolster bot detection against spoofed or otherwise manipulated video data.

Cite

@article{arxiv2607_27194,
  title={ VidMap: Exploiting Temporal Structure for Video-Based Structure-from-Motion },
  author={ Zador Pataki and Paul-Edouard Sarlin and Marc Pollefeys },
  journal={arXiv preprint arXiv:2607.27194},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.27194}
}

Read the full paper

• arXiv:2607.27194 (abstract)
• arXiv:2607.27194 (PDF)