DSP-SLAM++: A Unified Framework for Multi-Class, High-Fidelity Object SLAM in the Wild

Source: arXiv:2606.25953 · Published 2026-06-24 · By Ahmad Kourani, Ghina Daoud, Daniel Asmar, Imad Elhajj

TL;DR

DSP-SLAM++ addresses a critical trade-off in object-aware SLAM between real-time performance, multi-class object support, and high-fidelity, semantically coherent object reconstructions. Building on the prior DSP-SLAM framework, which used deep implicit shape priors but was limited to single classes and slow synchronous processing, this work introduces a novel asynchronous pipeline to decouple heavy reconstruction tasks from the main mapping thread. This enables multi-class reconstruction of detailed 3D meshes with up to 70% lower mapping latency. Meanwhile, the system integrates a dedicated sensor fusion pipeline for monocular fisheye cameras paired with LiDAR, solving the scale ambiguity challenge and leveraging wide-angle cameras common in automotive platforms. Evaluations on challenging real and benchmark datasets verify DSP-SLAM++ produces richer, geometrically complete multi-class maps with robust, real-time (25 Hz) performance outperforming DSP-SLAM in latency while maintaining comparable or improved tracking accuracy.

Key findings

• DSP-SLAM++ reduces maximum object processing latency by up to 70% compared to baseline DSP-SLAM under multi-class reconstruction (Table II).
• Asynchronous Reconstruction pipeline cuts mean mapping latency from >80ms to ~26ms in multi-class custom datasets, enabling stable 25 Hz system FPS.
• RGB-L sensor fusion module provides metric scale to monocular fisheye SLAM, avoiding the scale drift and tracking failures seen with monocular-only DSP-SLAM (Fig 6, Table I).
• Class-aware association with unique radii per object class improves detection-to-map matching robustness, reducing false matches and missed associations (Fig 3).
• Multiple independent DeepSDF decoders, one per class, avoid cross-class overfitting, enabling extensibility to multiple object classes without retraining entire model.
• Mask rectification for fisheye cameras prevents point cloud and semantic mask misalignment, critical for accurate reconstruction initialization (Fig 8).
• The system supports four classes (cars, buses, motorcycles, pedestrians), with qualitative robustness on large rigid objects and degradation on thin/articulated classes caused by sparse LiDAR data.
• DSP-SLAM++ demonstrates tracking ATE translational errors in the range 0.81–1.66 m and rotational errors around 0.77–4.10 degrees over multiple datasets, outperforming baseline in RGB-L monocular setups (Table I).

Methodology — deep read

1. Threat Model & Assumptions: The paper does not explicitly frame a formal security threat model, as this is a robotics SLAM system focusing on perception accuracy. Implicitly, the system assumes a real-world autonomous driving or robotic platform with a monocular fisheye camera and LiDAR sensor suite, operating in complex outdoor environments with multiple object classes. The adversarial scenarios would be environmental challenges like heavy occlusion, cluttered object scenes, and sensor noise/failure, but not deliberate adversaries.

2. Data: The main experimental evaluation uses a challenging custom in-house dataset collected by a vehicle with 4 fisheye cameras, a 32-line LiDAR, and GNSS/INS ground truth. Out of this, only the front fisheye camera and LiDAR are used. Additionally, established automotive datasets like KITTI (stereo and monocular), nuScenes (monocular), and CBNU (monocular fisheye) are utilized to show generalizability. Four object classes are considered: cars, buses, motorcycles, and pedestrians. Detectors used include YOLOv5 for 2D and PointPillars for 3D detection; 3D mesh shape priors are pretrained DeepSDF models trained on ShapeNet and other 3D repositories.

3. Architecture & Algorithm: DSP-SLAM++ extends the DSP-SLAM framework with three key innovations: 1) a multi-class detection and reconstruction pipeline, 2) an asynchronous reconstruction back-end to remove main thread bottlenecks, and 3) a fisheye+LiDAR sensor fusion front-end.

• Multi-class detection involves parallel 2D (YOLO segmentation masks + class IDs) and 3D (PointPillars bounding boxes + class IDs) detections per keyframe. A global bipartite greedy matching algorithm associates 2D masks to 3D boxes based on overlap and class consistency.
• Map objects are tracked via class-aware nearest neighbor association using class-specific spatial thresholds based on average object size to improve association accuracy.
• Shape reconstruction uses a modular multi-decoder DeepSDF architecture, with one separately pretrained decoder per object class to avoid cross-class interference.
• The asynchronous reconstruction pipeline decouples computationally heavy DeepSDF optimization from the local mapping thread. Reconstruction tasks are queued and processed on worker threads, with the mapping thread inserting placeholder objects and updating the map as reconstructions finish.
• The sensor fusion front-end modifies RGB-L depth extraction for fisheye cameras by rectifying sparse 2D feature points, enabling accurate association of LiDAR depth to visual features for metric scale recovery.

4. Training Regime: DeepSDF decoders are pretrained offline on canonical aligned 3D models per class from ShapeNet and online sources. Training details (epochs, optimizer, loss) are not thoroughly specified in the paper.

5. Evaluation Protocol: Evaluation metrics include Absolute Trajectory Error (ATE) in translation and rotation, map reconstruction quality assessed qualitatively via rendered 3D maps, and computational latency (detection latency, object processing latency, Keyframe Bundle Adjustment latency). Benchmarks are run across multiple datasets with 5 independent repeats. Comparisons are primarily between DSP-SLAM++ and the synchronous DSP-SLAM baseline. Ablations explore impact of asynchronous reconstruction and fisheye-LiDAR sensor fusion.

6. Reproducibility: Code is released open-source at the authors’ GitHub repository, enabling replication of the asynchronous SLAM pipeline. Dataset details for the custom dataset will be provided upon paper acceptance. Pretrained DeepSDF models availability is not specified.

Concrete example: For a given keyframe, YOLO and PointPillars generate candidate detections. These are matched by global bipartite assignment to form validated object observations. The system associates these with existing map objects using class-aware thresholds, or initializes new ones. Reconstruction tasks are pushed to a thread pool for DeepSDF-based shape optimization running asynchronously. Meanwhile, the local mapping thread immediately inserts temporary bounding box placeholders, enabling real-time tracking and mapping at 25 Hz without blocking on shape optimization completion. LiDAR points are rectified into virtual pinhole space to associate with visual features from the fisheye images, providing metric scale to the monocular SLAM backbone.

Technical innovations

• A global greedy bipartite matching algorithm for robust and efficient multi-class 2D-3D object detection association under lens distortion.
• A modular multi-decoder DeepSDF architecture that uses independent class-specific decoders to avoid overfitting and allow extensible multi-class high-fidelity shape reconstruction.
• An asynchronous object reconstruction pipeline that decouples heavy shape optimization from the main SLAM mapping thread to reduce latency by up to 70%, enabling real-time multi-class object SLAM at 25 Hz.
• A lightweight fisheye-LiDAR sensor fusion method that rectifies sparse 2D visual features instead of dense point clouds to efficiently provide metric scale to monocular fisheye SLAM.

Datasets

• Custom In-house urban driving dataset — size not specified, monocular fisheye camera + 32-line LiDAR — collected by authors
• KITTI — multiple sequences with stereo and monocular RGB cameras + LiDAR — public benchmark
• nuScenes — large-scale autonomous driving dataset with pinhole RGB cameras + LiDAR — public benchmark
• CBNU — monocular fisheye camera + LiDAR dataset — public dataset referenced by authors

Baselines vs proposed

• DSP-SLAM mono (without RGB-L) on KITTI 07: tracking failures and scale drift vs DSP-SLAM++ RGB-L: ATE translational error 0.85 m vs baseline failure (Table I)
• DSP-SLAM stereo on KITTI 07: ATE translational 0.83 m vs DSP-SLAM++ RGB-L 0.85 m (Table I)
• Synchronous Reconstruction (SR) vs Asynchronous Reconstruction (AR) on Custom multi-class dataset: Max object processing latency 566 ms vs 147 ms; Keyframe BA latency 2.5 frames vs 1.2 frames; system FPS 25.1 vs 25.4 (Table II)
• DSP-SLAM mono on nuScenes 127: failure vs DSP-SLAM++ RGB-L ATE translational ~0.85 m (Table I)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.25953.

Fig 1: DSP-SLAM++ generates consistent object-aware

Fig 2: DSP-SLAM++ system overview. MC stands for Multi-Class.

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• Reconstruction fidelity depends heavily on LiDAR point cloud density; sparse or occluded objects (especially thin articulated objects like pedestrians) yield incomplete or low-detail meshes.
• Performance and robustness are tightly coupled to the reliability of upstream 2D and 3D detectors (YOLO, PointPillars), which may struggle in highly cluttered or adverse scenarios.
• The system currently requires a LiDAR sensor to provide metric scale and depth data, limiting applicability to sensor suites without LiDAR or less expensive configurations.
• The asynchronous pipeline introduces latency for the final detailed mesh availability, which may impact some downstream tasks requiring immediate high-fidelity models.
• Authors acknowledge synthetic-to-real domain gap in pre-trained shape priors that may limit generalization without domain adaptation.

Open questions / follow-ons

• How to improve shape reconstruction quality for thin, articulated, or heavily occluded objects given sparse LiDAR data constraints?
• Can the asynchronous reconstruction paradigm be extended to dynamic object tracking and online shape update for moving objects?
• What methods could relax or remove LiDAR dependency for metric scale, enabling high-fidelity multi-class SLAM on monocular or stereo-only sensor setups?
• How might integration of object tracking and motion forecasting improve robustness and accuracy in dynamic environments?

Why it matters for bot defense

Although DSP-SLAM++ is primarily concerned with high-fidelity multi-class object reconstruction for autonomous driving and robotics applications, the techniques for asynchronous processing and multi-modal fusion have conceptual relevance to CAPTCHA or bot-defense systems reliant on scene understanding or 3D recognition. The asynchronous task offloading approach could inspire designs for scalable real-time verification systems that must handle complex object-level semantic interpretations without blocking user interactions. Additionally, robust multi-class object association methodologies and camera-LiDAR fusion approaches contribute insights for designing resilient sensing and mapping frameworks under distortion and occlusion — challenges analogous to bot recognition in adversarial or noisy visual domains. However, direct applicability to CAPTCHA technology is limited since this work addresses spatial mapping rather than challenge-response tests or adversarial interaction.

Cite

@article{arxiv2606_25953,
  title={ DSP-SLAM++: A Unified Framework for Multi-Class, High-Fidelity Object SLAM in the Wild },
  author={ Ahmad Kourani and Ghina Daoud and Daniel Asmar and Imad Elhajj },
  journal={arXiv preprint arXiv:2606.25953},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.25953}
}

Read the full paper

• arXiv:2606.25953 (abstract)
• arXiv:2606.25953 (PDF)