Semantic-weighted ICP for LiDAR Odometry: Class-Aware Residual Reweighting for Robust Scan Registration

Source: arXiv:2606.03905 · Published 2026-06-02 · By Vasco Carvalho, Tiago Barros, Urbano J. Nunes

TL;DR

This work addresses limitations in LiDAR odometry caused by dynamic objects, sparse geometric features, and semantically ambiguous structures in diverse and challenging environments. Traditional geometric-registration methods like ICP often degrade in accuracy when dealing with moving vehicles, vegetation, and off-road terrain where stable correspondences are scarce. Building upon prior semantic-aided methods that mostly perform binary filtering of points, the authors propose a semantic-weighted ICP where residuals are reweighted according to semantic class-based expected geometric stability. This soft weighting allows potentially informative but unstable classes to still contribute while reducing the influence of dynamic or unreliable classes.

Their approach extends SuMa++ by incorporating a semantic class-dependent weighting function λ(lk) applied to the residual weights in the point-to-plane ICP objective. Evaluations on two datasets, SemanticKITTI (urban, highway, rural) and RELLIS-3D (off-road/off-structured terrain), demonstrate that the semantic-weighted ICP improves pose estimation accuracy over the uniformly weighted baseline, particularly in off-road scenarios where geometry alone is insufficient. The authors also show that the optimal class weights vary significantly depending on environmental context, revealing the environment-dependent utility of semantic information for scan registration.

Key findings

• Semantic-weighted ICP reduces average translational drift from 0.759% to 0.695% and rotational drift from 0.310°/100m to 0.286°/100m on SemanticKITTI urban sequences (Table I).
• On off-road RELLIS-3D, semantic weighting reduces average translational drift from 41.04% to 25.77% and rotational drift from 41.46°/100m to 24.67°/100m (Table II).
• Hard removal of dynamic semantic classes (vehicles, pedestrians) does not consistently improve odometry and can degrade performance in sparsely structured scenes due to loss of geometric constraints.
• Soft weighting assigning low but nonzero weights to dynamic or unstable classes preserves useful information and improves scan matching robustness compared to equal weighting or hard filtering.
• Stable geometric classes like buildings, poles, and road provide the most reliable constraints and are assigned high weights to boost registration accuracy.
• Natural elements such as vegetation, terrain, and grass are assigned intermediate weights in off-road settings; zero or near zero weighting hurts performance because they form most of the observable geometry.
• Effectiveness of semantic residual weighting is environment dependent: stronger impact in off-road (RELLIS-3D) and urban structured environments, less consistent in highway or rural sequences where surfaces are large and repetitive.
• Using suMa++ baseline with uniform semantic class weights results in much higher rotational and translational drift errors compared to semantic-weighted ICP.

Threat model

Environmental dynamics such as moving vehicles, pedestrians, vegetation, and unstructured terrain induce geometric instability and spurious correspondences in LiDAR odometry. The adversary is effectively environment-induced noise and transient objects. The system assumes availability of semantic segmentation labels but no direct adversarial attack capabilities or attempts at malicious manipulation of sensor data.

Methodology — deep read

The paper addresses LiDAR odometry as a scan-to-map registration problem, estimating consecutive rigid transformations between point clouds. The key innovation is integrating semantic class-dependent residual weights into weighted point-to-plane ICP.

1. Threat model & assumptions: The adversary here is environmental dynamics such as moving vehicles, pedestrians, vegetation, and ambiguous geometry which degrade registration by inducing erroneous correspondences. The method assumes availability of semantic labels (via pretrained segmentation) and expects certain classes to be more geometrically stable than others.

2. Data: Two benchmark datasets are used. SemanticKITTI, providing semantic point cloud labels in urban, highway, and rural driving scenarios with ~20 labeled sequences. RELLIS-3D offers off-road sequences of unstructured terrain, vegetation, and sparse built structures. Both datasets have class annotations from semantic segmentation networks (RangeNet++).

3. Architecture / algorithm: They build on SuMa++ which uses surfel-based local maps and projective ICP. The core formulation minimizes a weighted point-to-plane ICP residual: E = ½ rᵀ W r, where r are residuals, W diagonal weight matrix. Weights wi combine a Huber loss, surfel stability indicator from the map γ(i), semantic compatibility Csem from label agreement, and their novel semantic class weighting λ(lk) for each semantic class.

4. Training regime: No explicit training of the weighting function is described. Weight values for each semantic class are heuristically assigned based on prior knowledge of geometric stability per class, tested in multiple weighting configurations.

5. Evaluation protocol: Odometry accuracy is measured by relative pose error (RPE) in translation % and rotation degrees/100m over trajectory segments. The baseline uses SuMa++ with uniform semantic weights. Variants with different class weightings are compared across multiple sequences from both datasets. Results report average errors per dataset sequence and qualitative trajectory plots versus ground truth.

6. Reproducibility: The method builds on publicly available baselines (SuMa++, RangeNet++). Exact class weights and source code availability are not explicitly stated, but the datasets and segmentation models are public. Hardware setup includes AMD Ryzen 7 with RTX 3060 GPU.

A concrete example: For a given scan, RangeNet++ predicts semantic labels per point. Residuals between the scan and the local surfel map points are computed along point-to-plane directions. Each residual is weighted using: wi = γ(i) · ρHuber(ri) · Csem(SPi, Smi) · λ(lk), where λ(lk) weights the semantic class of the scan point. The weighted ICP then optimizes the rigid transform minimizing these weighted residuals. The transform updates the global pose estimate, feeding into the next frame’s map update.

This soft weighting contrasts with prior hard filtering by dynamically emphasizing geometric stability according to semantic class, preserving weak but informative structures in off-road environments while attenuating dynamic object influence.

Technical innovations

• Integration of semantic class-dependent residual weighting into the ICP objective rather than hard filtering for LiDAR odometry, allowing soft suppression of unreliable classes.
• A semantic class-weighting function λ(lk) that encodes expected geometric stability as a continuous scalar weight per semantic class to modulate point influence during scan registration.
• Extension of surfel-based SuMa++ by combining surfel stability, semantic compatibility, Huber loss, and class-aware residual weights into a unified weighted ICP formulation.
• Empirical demonstration that soft, environment-adaptive semantic weighting outperforms uniform weighting or hard semantic rejection for scan registration across diverse urban and off-road datasets.

Datasets

• SemanticKITTI — 23 labeled driving sequences (urban, highway, rural) — public
• RELLIS-3D — off-road unstructured terrain sequences — public

Baselines vs proposed

• SuMa++ baseline (uniform semantic weights): SemanticKITTI Avg rot error = 0.310°/100m, trans error = 0.759%
• Semantic-weighted ICP best config: SemanticKITTI Avg rot error = 0.286°/100m, trans error = 0.695%
• SuMa++ baseline: RELLIS-3D Avg rot error = 41.46°/100m, trans error = 41.04%
• Semantic-weighted ICP best config: RELLIS-3D Avg rot error = 24.67°/100m, trans error = 25.77%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.03905.

Fig 1: Conceptual illustration of the proposed semantic class-weighted ICP.

Fig 2: Overview of the proposed LiDAR odometry pipeline. The raw LiDAR scan is processed by a semantic module (RangeNet++) to generate semantic

Fig 3: Qualitative overview of the evaluation datasets. The top row displays representative RGB camera frames, while the bottom row illustrates the

Fig 4: Qualitative trajectory comparison on representative SemanticKITTI [1] and RELLIS-3D sequences. The figure compares the ground truth trajectory,

Fig 5 (page 4).

Fig 6 (page 4).

Fig 7 (page 4).

Fig 8 (page 4).

Limitations

• The semantic class weights λ(lk) are heuristically chosen, no learning or optimization of weighting values reported.
• No adversarial or intentional robustness evaluation against malicious point cloud perturbations or spoofing attacks.
• Evaluation limited to known semantic classes and datasets; generalization to unseen environments or classes not studied.
• Effectiveness depends heavily on environment and class distribution; not universally better across all scenarios (notably highway and rural less consistent).
• No end-to-end learning of semantic segmentation and residual weighting jointly—segmentation errors can impact residual weighting.
• Exact reproducibility limited by lack of open-source code release or full hyperparameter details for weighting function.

Open questions / follow-ons

• Can the semantic class weighting function be learned adaptively or jointly with pose estimation from data rather than heuristically assigned?
• How robust is semantic-weighted ICP to segmentation errors or incomplete semantic label coverage?
• Could the approach be extended to incorporate temporal semantic stability estimation rather than static class-based weights?
• What is the impact of semantic weighting in tightly-coupled LiDAR-inertial odometry or full SLAM frameworks incorporating loop closures?

Why it matters for bot defense

For bot-defense or CAPTCHA systems relying on 3D LiDAR data or spatial registration tasks, this work demonstrates the benefit of semantic-aware weighting mechanisms that avoid rigid filtering of dynamic or ambiguous classes. Applying soft residual weighting based on class-dependent geometric stability can improve robustness of scan registration in noisy, cluttered, or dynamic scenes. Practitioners implementing lidar-based verification or environment mapping should consider utilizing semantic weighting to mitigate drift from transient objects rather than aggressively removing them, which may lose useful geometric information. The environment-dependent nature of optimal weighting suggests that cue weighting should be adapted or learned per domain rather than fixed.

Although developed for autonomous robotics, the principles of class-aware residual weighting in ICP registration can inspire more nuanced confidence weighting of sensor data in any system relying on point cloud alignment under noisy and dynamic conditions, such as physical bot-detection, occupancy verification, or pedestrian flow monitoring. However, the approach requires reliable semantic segmentation and a mechanism to assign semantic class weights reflective of geometric stability and scene characteristics. Integration into CAPTCHAs or bot detection pipelines would require aligned sensor modalities and semantic labeling capabilities.

Cite

@article{arxiv2606_03905,
  title={ Semantic-weighted ICP for LiDAR Odometry: Class-Aware Residual Reweighting for Robust Scan Registration },
  author={ Vasco Carvalho and Tiago Barros and Urbano J. Nunes },
  journal={arXiv preprint arXiv:2606.03905},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.03905}
}

Read the full paper

• arXiv:2606.03905 (abstract)
• arXiv:2606.03905 (PDF)