Self-Supervised Learning of Structured Dynamics from Videos

Source: arXiv:2607.21576 · Published 2026-07-23 · By Lukas Knobel, Andrew Zisserman, Yuki M. Asano

TL;DR

This paper addresses the challenge of disentangling camera motion and object motion in video understanding. Existing approaches often entangle these two sources of motion or require strong supervision, limiting robustness and scalability. The authors propose the Structured Dynamics Model (SDM), which operates on frozen pretrained image transformer features and explicitly separates temporal changes into a primary motion token capturing the dominant source of motion (camera or object), and a residual token capturing remaining dynamics. The model is trained via self-supervised future feature prediction with weak scene-level supervision on synthetic Kubric data and unlabeled real videos.

SDM is evaluated on ProbeMotion, a new benchmark suite spanning synthetic and real videos with diverse motion patterns. It outperforms baseline approaches using global CLS or average-pooled features and even strongly supervised 3D representation models on several tasks, despite substantially weaker supervision. The primary and residual tokens specialize according to motion regime, with the primary token capturing camera or object motion depending on static/dynamic camera conditions. These results demonstrate that frozen image backbones can be effectively remodeled into structured video-dynamics representations that disentangle key motion components with only weak supervision.

Key findings

• SDM reduces mean squared error (MSE) in Kubric camera motion regression from 0.96 (AVG-pool baseline) to 0.16 and Kubric object motion MSE from 0.93 to 0.19.
• On SSv2-110k action classification, SDM achieves 23.1% accuracy vs 2.3% for AVG-pool and 13.5% for CLS tokens.
• SDM outperforms DeltaTok, a strong self-supervised baseline trained 8× longer, on 5/7 ProbeMotion tasks (e.g., Kubric object motion MSE improvement of 0.16).
• SDM matches or outperforms strongly supervised models (VGGT, DA3, Pi3X) on multiple tasks, e.g., improves Kubric object motion MSE by 0.13 and YouTubeVOS object motion by 0.04 over best supervised baseline.
• Using temporal context (increasing sequence length from T=2 to T=4) reduces MSE on Kubric camera motion from 0.26 to 0.16 and increases SSv2-110k action recognition accuracy from 16.7% to 23.1%.
• Primary motion token p specializes in dominant scene change: predicts camera motion on dynamic-camera datasets and object motion on static-camera datasets; residual token r captures complementary dynamics.
• Weak scene-level labels (static scene or static camera) are critical to encourage the primary/residual decomposition during training.
• SDM is compatible with multiple frozen image backbones; DINOv2 and DINOv3 give best overall results.

Threat model

n/a — the paper does not explicitly define a security adversary model. The focus is on learning structured motion representations from video data using self-supervised and weakly supervised methods, rather than on adversarial threat or robustness evaluation.

Methodology — deep read

The authors assume an adversary interested in breaking motion representation but do not frame a security threat model explicitly here. The key challenge is disentangling two tightly coupled sources of video dynamics: camera motion and object motion, without strong geometric or 3D supervision.

The main data sources are:

• Synthetic Kubric dataset with 180k sequences and weak scene-level labels indicating static or dynamic camera and scene.
• Real unlabeled videos from Something-Something V2 (SSv2) with ~170k clips and DL3DV with 4k clips.

Training splits combine synthetic videos with weak scene-level labels and large-scale unlabeled real videos. The labels indicate static scene or static camera, used to guide loss function choices.

Feature extraction uses a frozen vision transformer backbone (DINOv2-B/14) to extract spatial feature maps per video frame. Intermediate features from all 12 layers are concatenated and projected to a common dimension (D=768).

The Structured Dynamics Model (SDM) operates on pairs of consecutive feature maps (f_{t-1}, f_t) to predict f_t from f_{t-1} in frozen feature space via future-feature prediction.

SDM decomposes temporal changes into two sequential steps:

1. Primary motion extraction and compensation:
   • A recurrent transformer-based motion extractor (ϕ_p) updates the primary motion token p_t by attending to (f_{t-1}, f_t) and previous p_{t-1}.
   • The primary predictor (ψ_p) applies the token p_t to compensate f_{t-1}, producing an intermediate compensated feature f_t^>.
2. Residual motion extraction and compensation:
   • Residual extractor (ϕ_r) computes residual motion token r_t from the discrepancy between f_t^> and f_t as well as r_{t-1}.
   • Residual predictor (ψ_r) refines f_t^> using r_t to produce the final prediction f_t^>>.

Both extractors are 4-block transformer decoders attending to frozen features with spatial-temporal rotary embeddings. Predictors are 2-block transformers attending to motion tokens.

Training loss is mean squared error in frozen feature space between predicted and actual features. The loss is selectively applied based on weak scene-level labels:

• For unlabeled or dynamic scenes, supervise final prediction f_t^>> only.
• For static scenes, supervise primary-compensated features f_t^> directly (residual stage bypassed).
• For static cameras, regularize primary stage to leave source features unchanged, pushing residual to capture scene dynamics.

Training uses AdamW on mixed batches from Kubric, SSv2, and DL3DV, totaling 200k iterations with batch size 128 on 4 Nvidia H100 GPUs (~11 hours). Input videos are resized to 224x224, sampled at 2 fps with 5-frame clips.

Evaluation uses linear probe classifiers or regressors on frozen motion tokens p and r extracted from held-out datasets unified in the ProbeMotion benchmark. These include camera motion, object motion, and action classification tasks across synthetic and real video sources.

Different tokens (p vs r) are probed separately based on dominant source of motion. Multiple ablations test token swapping, temporal context length, different backbones, and different loss variants.

Code release is not mentioned explicitly and datasets include public and proprietary splits.

Example workflow: Given a test video clip, extract frozen features per frame via pretrained DINOv2 backbone. For each consecutive frame pair, run the SDM model to compute primary and residual motion tokens predicting next frame features. Use the motion tokens at the final timestep as structured embeddings. Feed these embeddings to a linear classifier trained on labeled motion or action targets. Evaluate classification accuracy or regression MSE against ground truth labels for benchmark comparison.

Technical innovations

• Propose SDM, which structures temporal changes in frozen image features into sequentially extracted primary and residual motion tokens rather than a single latent.
• Use weak scene-level supervision (static camera / static scene indicators) to guide decomposition of motion into primary and residual components.
• Employ self-supervised future feature prediction on frozen pretrained image transformer representations, avoiding end-to-end video encoder training.
• Demonstrate that structured tokens specializing for camera vs object motion emerge naturally under this design and supervision.
• Introduce ProbeMotion, a diverse linear-probe evaluation suite spanning synthetic and real datasets evaluating disentangled camera/object motion and action recognition.

Datasets

• Kubric — 180k synthetic video sequences with weak scene-level labels — public synthetic dataset
• Something-Something V2 (SSv2) — 170k real video clips unlabeled for motion — public
• DL3DV — 4k real video clips unlabeled — source not specified (possibly public)
• ProbeMotion benchmark — aggregate evaluation suite constructed from Kubric, DL3DV, CameraBench, DAVIS2017 static-camera subset, YouTubeVOS static-camera subset, SSv2-110k subset

Baselines vs proposed

• AVG-pool feature differences baseline: Kubric camera motion MSE = 0.96 vs SDM = 0.16
• CLS token concatenation baseline: Kubric object motion MSE = 0.78 vs SDM = 0.19
• DeltaTok (self-supervised latent action model): SSv2-110k accuracy = 13.5% vs SDM = 23.1%
• Strongly supervised VGGT camera token: Kubric camera motion MSE = 0.03 vs SDM = 0.16 (close performance but VGGT uses much stronger supervision)
• Strongly supervised DA3 average feature: Static YouTubeVOS object motion MSE = 0.52 vs SDM = 0.71
• Strongly supervised Pi3X average feature: SSv2-110k accuracy = 19.8% vs SDM = 23.1%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.21576.

Fig 3: Stage-wise structured feature prediction. We visualize feature-map error to the target

Fig 4: Motion extrapolation. We repeatedly apply the last motion token from the observed frames

Fig 5: Latent motion swapping. We apply motion tokens from a source clip to the feature map of

Fig 4 (page 9).

Fig 5 (page 9).

Fig 6 (page 9).

Fig 7 (page 9).

Fig 8 (page 9).

Limitations

• Weak scene-level supervision only partially disentangles motion; separation quality depends on distribution and annotation noise in synthetic Kubric data.
• Residual motion token representations show high variability on some datasets (e.g., static DAVIS2017), indicating instability.
• Longer-term motion extrapolation drifts substantially, limiting usefulness for long-horizon prediction.
• Evaluation limited to linear probing; nonlinear downstream adaptation performance untested.
• Code release and full reproducibility not explicitly stated.
• Model size and compute requirements (4 H100 GPUs for 11 hours) may limit accessibility.

Open questions / follow-ons

• Can the primary/residual decomposition be further improved or generalized with stronger or alternative forms of weak supervision?
• How well do the learned structured motion tokens support nonlinear downstream tasks beyond linear probing?
• Can the architecture be extended to support long-horizon motion extrapolation without drift?
• What is the effect of noisy or domain-shifted camera/scene conditions on the disentanglement quality?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this paper highlights how structured disentanglement of motion sources (camera vs object) can be learned from unlabeled or weakly supervised data using pretrained image backbones. In a CAPTCHA or bot-detection context, understanding or modeling motion dynamics is often crucial for robust human-bot differentiation.

The SDM approach suggests a pathway to build interpretable and specialized motion descriptors without requiring expensive 3D annotations or end-to-end video training. Practitioners designing motion-based defense mechanisms or CAPTCHAs involving videos could consider incorporating structured tokens similar to SDM's primary and residual motion embeddings to better isolate meaningful object motion from camera-induced artifacts.

Moreover, the use of pretrained image transformers as frozen backbones offers scalability benefits and potential for transfer across domains. However, limitations such as drift in long-term extrapolation and dependency on weak scene-level labels highlight areas needing careful evaluation before direct application in adversarial or real-time systems.

Cite

@article{arxiv2607_21576,
  title={ Self-Supervised Learning of Structured Dynamics from Videos },
  author={ Lukas Knobel and Andrew Zisserman and Yuki M. Asano },
  journal={arXiv preprint arXiv:2607.21576},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.21576}
}

Read the full paper

• arXiv:2607.21576 (abstract)
• arXiv:2607.21576 (PDF)