Hilti-Trimble-Oxford Dataset: 360 Visual-Inertial Benchmark with Floor Plan Priors for SLAM and Localization

Source: arXiv:2607.06464 · Published 2026-07-07 · By Samuele Centanni, Yuhao Zhang, Yifu Tao, Julien Kindle, Frank Neuhaus, Tilman Koß et al.

TL;DR

This paper introduces the Hilti-Trimble-Oxford Dataset, a challenging visual-inertial benchmark designed for Simultaneous Localization and Mapping (SLAM) and floor plan-referenced localization in active construction environments. The dataset includes 30 sequences collected over eight months on a seven-floor building under construction, using a consumer-grade 360° camera with embedded IMU, with accurate ground truth trajectories from a high-quality LiDAR-inertial SLAM system rigidly attached to the camera. The dataset captures realistic difficulties such as dynamic scenes with moving workers, low-light conditions, aggressive motions, and repetitive textures, representative of construction site complexities. It includes synchronized 360° video, IMU data, and simplified 2D floor plans, enabling benchmarking of algorithms in both unconstrained SLAM and localization within architectural floor plans.

Beyond data release, the authors organized the Hilti x Trimble Challenge 2026, attracting 62 teams for SLAM and 22 for localization. The challenge results reveal that top visual-inertial SLAM solutions built on state-of-the-art open frameworks (e.g., OKVIS2-X) achieved 3D localization errors as low as 9 cm after global optimization. For floor-plan-based localization, the best methods combining visual-inertial odometry with semantic segmentation to extract stable wall features achieved average 2D localization errors around 24 cm. The benchmark highlights the difficulty of precise localization relative to floor plans compared to unconstrained SLAM in realistic, evolving built environments, emphasizing the need for further research. The dataset, ground truth, evaluation code, and challenge leaderboard are publicly available for ongoing algorithm development and evaluation.

Key findings

• The dataset comprises 30 visual-inertial sequences recorded over 8 months, spanning 7 floors and capturing diverse conditions (low-light, clutter, aggressive motion).
• Top SLAM solutions based on OKVIS2-X achieved a total score of 2410/2500 points with mean 3D position errors as low as ~9 cm (Fig. 6).
• Localization methods using semantic segmentation to extract walls and BEV projection achieved an average RMSE of approximately 24 cm, scoring 2196.3/2400 points.
• SLAM track saw much higher participation (62 unique teams) than localization (22), indicating higher maturity in the unconstrained trajectory estimation task.
• Dynamic initialization starting in motion remains challenging for many SLAM methods, addressed through specialized delayed or adaptive initialization strategies.
• Low-light and texture-poor underground floors consistently caused larger errors in both SLAM and localization tracks (Fig. 6).
• The floor plans used do not always reflect on-site changes (e.g., missing partitions or ducting in early stages), requiring localization despite scene-to-plan discrepancies (~2 cm average deviations).
• Use of line features alongside point features improved robustness in repetitive, texture-sparse indoor construction environments.

Methodology — deep read

1. Threat Model & Assumptions: While not explicitly formulated as a security problem, the benchmark assumes algorithms must operate in a challenging active construction site environment with dynamic obstacles (workers), lighting variation, scene evolution, and repetitive geometry complicating reliable visual-inertial pose estimation and floor plan alignment.

2. Data: The dataset consists of 30 sequences averaging 180 seconds each, acquired over 8 months at a real construction site with evolving structures across seven floors and two underground parking levels. Data includes two synchronized fisheye camera streams from an Insta360 ONE RS 1-Inch 360 camera at 30 Hz (front and rear hemispheres), and 6-axis IMU measurements at 1000 Hz, formatted as ROS 2 bag files. The sequences vary in difficulty factors: clutter, dynamic initialization (starting in motion), low-light, and aggressive 6-DoF motions causing motion blur and rolling shutter challenges. Floor plans are provided as binary PNG images (1 px = 1 cm) and DXF files, showing structural walls and columns; two variants exist with/without windows. Ground truth camera trajectories come from a high-accuracy LiDAR-inertial SLAM system (Hesai XT32M2X LiDAR plus tactical-grade IMU) rigidly mounted with the camera, temporally synchronized using cross-correlation of inertial signals.

3. Architecture/Algorithm: The winning SLAM approach (ACDC-VSLAM) builds on OKVIS2-X visual-inertial odometry, employing both point and line features with optimized feature distribution per image, a local bundle adjustment re-weighting constraints by observation reliability and depth, and a two-stage adaptive loop closure module that selects the best pose constraints (SE(3), Sim(3), or 4-DoF). For localization, the top method combined OKVIS2-X odometry with Z-FLoc, extracting wall points from semantic segmentation to project into a bird's-eye-view map, matched via RANSAC and refined with non-linear optimization, followed by pose graph refinement.

4. Training Regime: Not detailed as challenge participants used existing algorithms or variants. Evaluation sequences were split into 5 held-out runs per track for validation; these were excluded from final scoring to prevent overfitting.

5. Evaluation Protocol: Trajectories submitted must cover at least 99% of ground truth poses (or score zero). SLAM trajectories undergo rigid alignment with the ground truth using the Kabsch algorithm, and 3D position errors are computed. Localization trajectories must be reported in floor plan coordinates with no alignment, evaluated via 2D Euclidean distance ignoring height. An exponential decay scoring function maps position error in meters to a score with parameters making 0 m error = 100 points and 10 m error ~ 1 point. Final scores sum trajectory-level exponentially weighted position accuracies. The evo toolbox was used for evaluation.

6. Reproducibility: The full dataset, including raw sensor data, floor plans, ground truth trajectories, and evaluation code were publicly released alongside the open Hilti x Trimble Challenge 2026 benchmark. The challenge attracted global participation with detailed leaderboards, fostering reproducible comparisons.

Concrete example end-to-end: A sequence from one floor with significant dynamic workers and low-light is captured with the Insta360 camera and IMU; the LiDAR-inertial system captures ground truth trajectories simultaneously. A team uses OKVIS2-X to run visual-inertial odometry on both fisheye cameras, extracting point and line features, performing local bundle adjustment, leveraging loop closures to reduce drift, then aligns their trajectory to ground truth via Kabsch for scoring. For localization, they segment walls from camera images, project features into a 3D wall point cloud, align this with the provided 2D floor plan using RANSAC, apply pose graph optimization, and submit their final estimated poses in floor plan coordinates for evaluation.

This dataset and evaluation framework enable benchmarking of multi-camera 360° visual-inertial SLAM and challenging floor-plan-based localization algorithms in realistic, unstructured, and evolving indoor industrial environments.

Technical innovations

• Introduction of a large-scale, long-term 360° visual-inertial dataset capturing evolving multi-floor construction environments with real-world challenges (dynamic occlusions, lighting, repetitive structures) not present in prior datasets.
• Provision of precise ground truth trajectories using tightly coupled LiDAR-inertial SLAM with extrinsic calibration and temporal alignment for validation of consumer-grade camera and IMU tracking.
• A benchmark task combining unconstrained visual-inertial SLAM with evaluation of floor plan-referenced localization referencing simplified 2D floor plans rather than complex BIM models.
• Demonstration through challenge results that semantic segmentation-based extraction of stable structural elements (walls) improves localization accuracy relative to floor plans compared to purely geometric matching.

Datasets

• Hilti-Trimble-Oxford Dataset — 30 sequences over 8 months, 7 floors — Active construction site, publicly released at https://hilti-trimble-challenge.com/dataset-2026

Baselines vs proposed

• ACDC-VSLAM (SLAM track, winner): final score = 2410/2500 with mean 3D position errors ~9 cm
• Joint ETH-THU-Stuttgart team (Localization track, winner): average RMSE approx. 24 cm, score = 2196/2400
• Other top SLAM baselines based on OKVIS2-X, OpenVINS, ORB-SLAM3, and √VINS with varied accuracy; no single detailed metric comparison reported beyond challenge leaderboard summaries

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.06464.

Fig 1: The Hilti-Trimble-Oxford Dataset, consisting of 360-degree

Fig 2: Visualization of the multi-camera calibration. Extrinsic

Fig 3 (page 1).

Fig 4 (page 1).

Fig 3: Types of provided floor plans.

Fig 4: Data collection platform: rigidly connected V&R LiDAR-

Fig 5: A ground truth trajectory (blue) with LiDAR SLAM map

Fig 6: Average error of the top three teams for each sequence in

Limitations

• Floor plans provided do not reflect ongoing construction changes, causing localization to face mismatches between recorded scenes and architectural plans; this makes challenges realistic but introduces uncertainty.
• Some minor residual errors in ground truth alignment pipeline due to architectural deviations or errors in floor plans (e.g., partitions with slightly different lengths) exist but considered acceptable for benchmark scale.
• The camera system does not have hardware synchronization with LiDAR; temporal alignment is done in software via inertial cross-correlation, which may limit synchronization precision.
• The dataset focuses on visual-inertial data from a 360 camera and excludes LiDAR, limiting sensors tested to low-cost consumer-grade hardware though ground truth comes from LiDAR.
• While the challenge attracted many submissions, no adversarial robustness evaluation or deliberate attack scenarios on SLAM/localization were presented.
• Localization methods struggle notably in underground floors with low light and sparse texture, which remain unsolved challenges.

Open questions / follow-ons

• How to improve localization accuracy under significant discrepancies between as-built environments and outdated or simplified floor plans, particularly in dynamic and evolving construction contexts?
• What novel sensor fusion or semantic mapping approaches could better handle low-light, cluttered, and repetitive texture indoor scenes common in construction?
• Can long-term consistency and multi-session visual-inertial SLAM be improved to handle months-long construction site changes and structural evolution?
• How to efficiently incorporate additional map priors (e.g., windows, doors, BIM data) without overwhelming computational complexity for practical construction monitoring?

Why it matters for bot defense

This dataset and benchmark provide a valuable testbed for bot-defense engineers interested in robust visual-inertial pose estimation and scene localization under adverse and dynamic real-world conditions. Insights from the challenge results show that combining semantic segmentation with geometric alignment improves localization robustness even when reference floor plans do not perfectly match the operational scene, a useful principle for designing systems resilient to environmental variations or attempts at spoofing with altered spatial layouts. The dataset’s multi-camera 360° setup and inclusion of IMU data offer guidance on sensor modalities useful for robust environment mapping, which is essential to distinguish legitimate human interaction from automated bot or replay attacks that rely on static camera inputs. The long-term data captures evolving scene properties, encouraging research into temporal consistency and adaptation mechanisms crucial when combating persistent or adaptive automated attacks in physical spaces. However, the challenge highlights that even state-of-the-art visual-inertial SLAM and localization methods still exhibit substantial errors under difficult lighting and clutter, underlining the necessity of including multiple complementary defenses for CAPTCHA and bot mitigation applications.

Cite

@article{arxiv2607_06464,
  title={ Hilti-Trimble-Oxford Dataset: 360 Visual-Inertial Benchmark with Floor Plan Priors for SLAM and Localization },
  author={ Samuele Centanni and Yuhao Zhang and Yifu Tao and Julien Kindle and Frank Neuhaus and Tilman Koß and Aryaman Patel and Michael Helmberger and Emilia Szymańska and Torben Gräber and Maurice Fallon },
  journal={arXiv preprint arXiv:2607.06464},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.06464}
}

Read the full paper

• arXiv:2607.06464 (abstract)
• arXiv:2607.06464 (PDF)