DF$^3$: World Modeling via Decoder-Free Feature Forecasting in Autonomous Navigation

Source: arXiv:2608.02428 · Published 2026-08-03 · By Jiaming Chen, Guoan Xu, Aoshen Huang, Haozhuo Zhang, Yang Li, Wei Pan

TL;DR

This paper addresses the critical challenge of forecasting future scene states from video sequences for autonomous robotic navigation. Prior pixel-level generative methods suffer from heavy computation and task-irrelevant details, while latent feature forecasting approaches rely on expensive decoders to convert predicted features into downstream predictions, causing inefficiency. The authors propose DF3, a Decoder-Free Feature Forecasting framework that performs future state prediction purely within the latent space of a frozen vision foundation model (ViT), eliminating the need for any decoder. DF3 injects learnable spatial prediction queries into the terminal blocks of the frozen ViT to directly produce future features. A novel Motion-Aware Context Fusion (MACF) module integrates coarse flow warping and fine cross-correlation of latent features to align and forecast future frames. Downstream task queries then probe these forecasted features for predictions such as semantic segmentation, all within the encoder and without decoder overhead.

Extensive experiments on the Cityscapes benchmark and zero-shot deployment in a quadruped robot simulator show DF3 achieves competitive accuracy to state-of-the-art latent forecasting methods but requires 36% fewer GFLOPs, 70% lower latency, 67% less peak GPU memory, and 41% fewer parameters. Qualitative results validate robust motion modeling and cross-embodiment generalization. The work provides a highly efficient, lightweight world model suitable for real-time perception and control in autonomous navigation settings.

Key findings

• DF3 achieves 69.9% mIoU and 68.7% MO-mIoU on short-term (T+1) Cityscapes forecasting, slightly below DINO-Foresight's 71.8% and 71.7%, but competitive in mid-term forecasting as well.
• DF3 reduces computational cost by 36% GFLOPs, cuts inference latency by 70% (to 292 ms/frame), reduces peak GPU memory by 67% (3.1 GB), and compresses model parameters by 41% compared to DINO-Foresight.
• The unified motion-aware context fusion (MACF) combining flow warping and latent cross-correlation improves short-term mIoU from 63.6% (warp only) and 65.7% (xcorr only) to 69.9%.
• Increasing the cross-correlation search radius r from {1} to {1, 2, 4, 8} steadily improves mIoU from 66.3% to 69.9%, demonstrating better capture of fast motion.
• Decoder-free design completely eliminates the need for heavy spatio-temporal or task-specific decoders by performing both forecasting and task predictions within a frozen ViT encoder.
• Zero-shot deployment on MATRiX quadruped robot simulator (trained only on Cityscapes driving data) showed spatially aligned trajectory predictions, confirming cross-embodiment generalization.
• Forecasted semantic segmentation features successfully model dynamic object motions and static scene layout despite operating in downsampled latent space without decoder refinement.
• Baseline ablations show naive concatenation or global attention fusion underperform explicit motion modeling techniques.

Threat model

n/a — This paper focuses on efficient and accurate latent-space future state forecasting for autonomous navigation rather than security adversaries or threat mitigation.

Methodology — deep read

1. Threat Model & Assumptions: The adversary context is indirect here—mainly the challenges are around modeling temporal dynamics from static-image pre-trained vision transformers for future state prediction in autonomous navigation. The model operates on past video frames and predicts future latent features without decoder overhead or fine-tuning the backbone, assuming no direct adversarial interference.

2. Data: Training and evaluation are performed on the Cityscapes dataset containing 2,975 training, 500 validation, and 1,525 test video sequences. Each sequence has 30 frames with the 20th frame annotated for semantic segmentation. Frames are resized and processed via sliding windows at 768x768 resolution.

3. Architecture & Algorithm: The core backbone is a frozen DINOv3-pretrained Vision Transformer (ViT-B/16). For each historical frame, patch tokens and prefix tokens (global and register tokens) are extracted from an intermediate layer (L1) of the ViT as context features.

A set of learnable prediction queries (Qpred) are injected into terminal ViT blocks (L2) concatenated with context tokens to forecast future frame features. Causal asymmetric attention masks prevent leakage, allowing prediction queries to attend to context but not vice versa.

The Motion-Aware Context Fusion (MACF) module refines predicted spatial queries by combining two branches: (a) a flow-based warp branch where a latent flow field is predicted to warp recent features; and (b) a cross-correlation branch that computes local cosine similarity between consecutive frame features within a variable radius neighborhood to extract semantic motion vectors. These are fused via a learned sigmoid gating mechanism.

Separate learnable task queries (Qtask) are injected into the terminal blocks along with forecasted future features to produce downstream task outputs (e.g. semantic segmentation masks), via lightweight MLP heads, all within the encoder's latent space.

4. Training Regime: The frozen encoder backbone is never fine-tuned. Models are trained for 400 epochs on 4 A100 GPUs using AdamW optimizer with base learning rate 5e-5 and cosine annealing schedule with warmup. The loss combines cosine similarity and Huber loss terms between predicted and true latent frame features, applied at both patch and prefix token levels.

5. Evaluation Protocol: Metrics include semantic segmentation mean Intersection-over-Union (mIoU), Moving Object mIoU (MO-mIoU), GFLOPs, inference latency (ms/frame), peak GPU memory (GB), and number of parameters. Baselines include the DINO-Foresight latent forecasting model and an oracle upper bound using ground-truth features.

Ablations study the impact of different fusion strategies (concat, attention, warp only, cross-corr only, warp+x-corr) and cross-correlation search radius settings. Qualitative evaluation involves visualizing segmentation and robotic navigation trajectories in the MATRiX simulator under zero-shot conditions.

6. Reproducibility: Code is not explicitly mentioned as released. Training uses publicly available Cityscapes dataset and pre-trained DINOv3 ViT backbone. Experimental details including architecture, hyperparameters, and metrics are thoroughly described allowing reproducibility.

Concrete Example: Given 5 historical video frames, their patch and prefix tokens are extracted by the frozen DINOv3 ViT at layer L1. Learned prediction queries are concatenated and injected into terminal block L2, attending over context tokens using an asymmetric mask. The MACF module refines spatial queries by predicting latent flow to warp recent features and then applying cross-correlation with prior frame features over a local radius window to estimate semantic motion. The fused update is gated and added to queries yielding future frame token features. Task queries then attend over these forecasted features to produce semantic segmentation maps. Loss is computed with cosine similarity and Huber loss against true future latent features to train prediction and task queries without modifying backbone weights.

Technical innovations

• Introduction of Decoder-Free Feature Forecasting (DF3) framework performing future state prediction entirely within the frozen vision foundation model latent space, removing the need for any decoder.
• Design of a novel Motion-Aware Context Fusion mechanism combining coarse latent flow warping and fine-grained local latent cross-correlation to explicitly model temporal dynamics without backbone fine-tuning.
• Use of learnable spatial prediction queries and task queries injected directly into terminal Vision Transformer blocks with asymmetric attention masking to separate context encoding and forecasting.
• Demonstration of zero-shot cross-embodiment transfer of learned latent dynamics from driving scenes to quadruped robot navigation, without additional adaptation.

Datasets

• Cityscapes — 5,000+ annotated video sequences with semantic labels for urban driving — publicly available

Baselines vs proposed

• DINO-Foresight [13]: Short-term mIoU = 71.8% vs DF3: 69.9%
• DINO-Foresight MO-mIoU = 71.7% vs DF3: 68.7%
• DINO-Foresight FLOPs = 2256.07 GFLOPs vs DF3: 1440.63 GFLOPs (-36%)
• DINO-Foresight Latency = 971.1 ms vs DF3: 292.4 ms (-70%)
• DINO-Foresight Peak GPU Memory = 9.5 GB vs DF3: 3.1 GB (-67%)
• DINO-Foresight Parameters = 302.5 MB vs DF3: 177.0 MB (-41%)
• Oracle Upper Bound mIoU = 79.8%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.02428.

Fig 1: Conceptual comparison of forecasting paradigms. (a) Generative

Fig 2: The overall pipeline of the proposed DF3 framework.

Fig 3: The visualization results of DF3 on Cityscapes.

Fig 4: Qualitative deployment results in the MATRiX simulator. The left column illustrates the simulation environment and the zsl-1 quadruped robot.

Limitations

• DF3 relies on a frozen vision encoder; it does not fine-tune backbone weights to potentially improve temporal modeling capacity.
• The model currently supports observation-only forecasting without action-conditioned future prediction needed for closed-loop control.
• Forecasted features exhibit spatial over-smoothing, resulting in coarse object boundaries and less sharpness in thin or fine structures.
• Cross-correlation window size tuning is required to balance capturing fast motion vs noise sensitivity, potentially limiting generalization across domains.
• Zero-shot deployment shows spatial misalignments due to domain gap from training on driving data and testing on quadruped locomotion.
• Code and pretrained weights release status are not stated, limiting reproducibility.

Open questions / follow-ons

• How to extend decoder-free forecasting for action-conditioned world modeling to enable closed-loop agent planning?
• Can lighter vision backbones or alternative architectures maintain performance while further reducing computational overhead?
• Is it possible to fine-tune the frozen encoder or adaptively modulate weights to improve temporal alignment without losing generalization?
• How well does DF3 generalize to other modalities or environments beyond urban driving and quadruped locomotion?

Why it matters for bot defense

DF3's approach to eliminate decoders by forecasting entirely within frozen encoder feature space offers a compelling paradigm to reduce computation and latency in real-time downstream tasks. For bot-defense or CAPTCHA scenarios involving video or temporal visual data, leveraging decoder-free latent forecasting could enable faster, more efficient verification by directly predicting future states or challenge-relevant features without expensive pixel-space reconstruction or heavy decoding. The motion-aware fusion injecting temporal cues into static-image pretrained transformers also suggests novel ways to repurpose foundation models for temporal understanding without retraining large models. However, the lack of action conditioning limits closed-loop interactive use cases. Practitioners should weigh DF3's gain in efficiency against slightly reduced accuracy and coarser spatial fidelity depending on CAPTCHA task sensitivity.

Cite

@article{arxiv2608_02428,
  title={ DF$^3$: World Modeling via Decoder-Free Feature Forecasting in Autonomous Navigation },
  author={ Jiaming Chen and Guoan Xu and Aoshen Huang and Haozhuo Zhang and Yang Li and Wei Pan },
  journal={arXiv preprint arXiv:2608.02428},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.02428}
}

Read the full paper

• arXiv:2608.02428 (abstract)
• arXiv:2608.02428 (PDF)