DDStereo: Efficient Dual Decoder Transformers for Stereo 3D Road Anomaly Detection

Source: arXiv:2606.24805 · Published 2026-06-23 · By Shiyi Mu, Zichong Gu, Zhiqi Ai, Yilin Gao, Shugong Xu

TL;DR

This paper addresses two critical challenges in stereo-based 3D object detection for autonomous driving: achieving real-time inference speeds and enabling robust open-set detection of unknown obstacles. While stereo methods generally outperform monocular detectors in accuracy, they tend to run too slowly for immediate safety-critical usage. Additionally, most prior stereo detectors assume a closed-set label set and struggle to detect novel out-of-distribution (OoD) objects. To overcome these gaps, the authors propose DDStereo, a Dual-Decoder Stereo Transformer architecture that decouples binary foreground localization (category-agnostic) from multi-class 3D attribute regression, sharing object queries between these decoders to unify detections and facilitate anomaly scoring. A compact disparity feature extractor and efficient Transformer decoder blocks further improve speed. Empirical evaluation on the KITTI Stereo 3D benchmark and the KITTI-AR-OoD dataset demonstrates state-of-the-art accuracy under both closed-set and open-set settings with inference times reduced to 23.5 ms per frame—surpassing prior stereo detectors by a wide margin and matching real-time monocular methods. The model also exhibits strong generalization for pedestrian and cyclist classes and outperforms existing open-set stereo 3D detection approaches such as S3AD in both accuracy and speed. Overall, DDStereo presents an effective and practical step toward real-time, open-world stereo vision-based 3D detection for autonomous driving.

Key findings

• DDStereo runs in 23.5 ms per frame, achieving real-time performance and surpassing prior stereo detectors such as TS3D (88 ms), RTS3D (39 ms), and YOLOStereo3D (80 ms).
• On the KITTI 3D detection benchmark, DDStereo achieves state-of-the-art AP3D on Moderate and Hard settings for the Car class with 43.97% and 36.16%, respectively.
• DDStereo requires only 62.65 GFLOPs and 19.6M parameters, about half the size of monocular models like MonoDETR and one-fifth of YOLOStereo3D, enabling efficient deployment.
• Open-set evaluation on KITTI-AR-OoD shows DDStereo attains 74.23% AP3D, 83.01% APBEV, and 98.36% AP2D for OoD objects at R11 recall, outperforming S3AD especially in bird’s-eye-view localization.
• Shared object queries aligning foreground and 3D attribute decoders improve open-set AP OoD 3D accuracy by ~27% compared to independent queries.
• Dual-decoder architecture for separate foreground detection and 3D regression improves AP OoD 3D accuracy by 11.72% over a single-decoder model.
• Grid sample-based depth extraction from predicted 2D centers boosts AP OoD 3D accuracy by 8.7% relative to average-based depth aggregation.
• Using sigmoid normalization for anomaly scoring surpasses MaxLogit and MSP methods by 2.9% to 10.4% AP OoD 3D.

Threat model

The adversary corresponds to unknown or previously unseen objects (out-of-distribution obstacles) that may appear on the road, such as novel vehicles, debris, or construction equipment. The adversary is passive (i.e., naturally occurring or rare obstacles) and does not manipulate sensor inputs but challenges the detector's ability to identify objects beyond known categories. The model must detect these unknown objects using only stereo visual input, without external prompts or queries. The system assumes no access to external side-channel data or deliberate attacks on sensor integrity.

Methodology — deep read

The authors address stereo 3D detection under an open-world setting where the adversary corresponds to unknown or novel objects not seen during training, such as unseen road anomalies or debris; detection must operate in real-time and recognize OoD instances without additional input prompts.

They train and test primarily on the KITTI Stereo benchmark for known classes (Car, Pedestrian, Cyclist) with 3K training stereo pairs and evaluate open-set detection on the KITTI-AR-OoD dataset, which contains out-of-distribution categories uniformly labeled as 'OoD'. Ground truth includes 3D box annotations and 2D bounding boxes for stereo pairs.

The DDStereo architecture consists of: (1) a ResNet backbone extracting multi-scale 2D features from the left image, (2) a disparity feature extractor that builds correlation volumes by shifting right image features horizontally to encode stereo depth cues, and (3) two Transformer decoders sharing learnable object queries. One decoder performs binary foreground detection on disparity features using a DETR-style Transformer decoder with deformable attention, predicting 2D foreground boxes and objectness scores. The other decoder predicts 3D attributes (size, orientation, class scores for known categories, 2D boxes parameterized by projected 3D centers) from left-image features.

Depth is separately predicted via a lightweight convolutional U-Net upsampling fused multi-scale stereo correlation volumes to a high-resolution depth map at 1/4 scale. Object depth estimates are sampled at the projected 2D centers from the 3D detection decoder using grid sampling.

Both decoders use the same object query matrix to ensure spatial correspondence and enable anomaly scoring via a Maximum Normal Probability minus Foreground confidence (MNPF) metric, leveraging sigmoid-normalized classification scores to identify OoD objects without text prompts or anchors.

The model is trained end-to-end with a combined 2D foreground loss (binary classification and bounding box regression), a 3D detection loss (multi-class classification and 3D regression), and a disparity supervision loss. Hungarian matching is applied independently on 2D and 3D prediction sets during training for target assignment.

Training uses Adam optimizer, batch size 16, initial lr=0.0002 decayed twice, over 165 epochs on a single NVIDIA RTX 4090 GPU.

Evaluation measures standard AP metrics for 3D, BEV, and 2D bounding boxes at IoU thresholds, comparing both closed-set and open-set performance with baselines. Open-set metrics treat OoD as an extra class with anomaly scoring. Ablations isolate query sharing, decoder splits, depth sampling, and OoD scoring methods.

Reproducibility is supported by detailed architectural diagrams, loss formulations, and training hyperparameters, though no code or pretrained weights were publicly released at the time of the paper.

Technical innovations

• A dual-decoder Transformer architecture separating disparity-based binary foreground detection from left-image-based multi-class 3D attribute regression with shared object queries enables unified open-set 3D detection.
• Introduction of a compact multi-scale stereo correlation volume for efficient disparity feature extraction that avoids heavy anchor-based or cost-volume designs.
• Object-level depth map prediction at high (1/4) resolution combined with grid-sampling-based extraction of depth at projected 2D centers simplifies depth estimation and improves accuracy.
• Novel MNPF open-set confidence scoring metric calculated by subtracting maximum known-class probability from foreground prediction, optimized under sigmoid normalization for robust OoD detection.
• Shared learnable queries across dual decoders to maintain spatial and semantic consistency without the need for NMS or anchor matching, enabling end-to-end training and inference.

Datasets

• KITTI Stereo Benchmark — ~3,000 stereo training pairs + test server — public
• KITTI-AR-OoD — Out-of-distribution annotated stereo test set for open-set detection — public

Baselines vs proposed

• TS3D: runtime = 88 ms vs DDStereo runtime = 23.5 ms
• RTS3D: AP3D moderate (Car) = 37.38% vs DDStereo = 43.97%
• S3AD (open-set): AP OoD 3D at R11 = 74.35% vs DDStereo = 74.23% (similar recall) but DDStereo achieves higher APBEV of 83.01% vs 76.39%
• YOLOStereo3D: GFLOPs = 177.82 vs DDStereo GFLOPs = 62.65; Params 107.6M vs 19.6M
• MonoDETR: AP3D moderate = 16.47% vs DDStereo stereo AP3D moderate = 43.97% (Car)
• Ablation shared queries: open-set AP OoD 3D = 78.09% vs independent queries = 51.19%
• Ablation dual-decoder: AP OoD 3D = 78.09% vs w/o dual-decoder = 66.37%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.24805.

Fig 2: Compare with other Open-set 3D object detection methods.

Fig 1: Comparison of inference speed and accuracy with exist-

Fig 3: Overview of the proposed DDStereo architecture.

Fig 4: Computation process of the stereo correlation volume

Fig 5: Multi-scale disparity fusion and depth map prediction.

Fig 6: Visualization results with 3D boxes and BEV map.

Fig 7 (page 7).

Fig 8 (page 7).

Limitations

• Training data size is relatively small (~3K stereo pairs), limiting performance on easier subsets and possibly the full generalization potential.
• No adversarial robustness or attack resilience evaluation, important for safety-critical autonomous driving scenarios.
• Evaluation datasets focus on KITTI and KITTI-AR-OoD; more extensive real-world benchmarks representing diverse weather, lighting, or traffic conditions are not tested.
• Depth estimation relies on grid sampling from predicted 2D centers, which may degrade with severe occlusions or truncated objects.
• Open-set detection treats all unknown categories as a single OoD class without fine-grained recognition or vocabulary extension.
• Code, pretrained models, and training details like random seed not released, limiting immediate reproducibility.

Open questions / follow-ons

• How to extend DDStereo's open-set detection to handle fine-grained categorization or open-vocabulary recognition beyond binary unknown detection without incurring large latency?
• Can the dual-decoder Transformer framework integrate temporal stereo sequences or multi-view inputs to enhance depth accuracy and OoD detection stability?
• What strategies can improve robustness and generalization of open-set detection under domain shift, adverse weather, or sensor noise conditions?
• Is it feasible to leverage large-scale synthetic stereo datasets combined with self-supervised or foundation model pretraining to improve zero-shot OoD detection?

Why it matters for bot defense

From a bot-defense and CAPTCHA perspective, this paper's approach holds insights for efficient real-time anomaly detection that generalizes beyond predefined categories. The dual-decoder separation of foreground segmentation and multi-class attribute prediction with shared queries offers a promising paradigm for detecting unknown threats or novel bot behaviors that do not conform to training categories, while maintaining low latency critical in real-time applications.

Furthermore, the object-level anomaly scoring based solely on visual inputs, without reliance on external textual or query prompts, mirrors the practical constraints in automated detection pipelines needing immediate and reliable identification of unfamiliar inputs. The disparity-driven foreground localization leveraging stereo geometry as an additional signal contrasts with purely appearance-based methods and suggests multi-modal feature alignment may enhance generalization for security tasks.

Overall, the architectural innovations and open-set confidence formulations can inspire more adaptive bot and anomaly detection systems that do not rely on exhaustive training on all possible attack types or categories, improving robustness against novel attacks with computational efficiency.

Cite

@article{arxiv2606_24805,
  title={ DDStereo: Efficient Dual Decoder Transformers for Stereo 3D Road Anomaly Detection },
  author={ Shiyi Mu and Zichong Gu and Zhiqi Ai and Yilin Gao and Shugong Xu },
  journal={arXiv preprint arXiv:2606.24805},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.24805}
}

Read the full paper

• arXiv:2606.24805 (abstract)
• arXiv:2606.24805 (PDF)