CLIFE: Camera-LiDAR Fusion Framework for Edge-Deployable Roadside VRU Perception

Source: arXiv:2607.16154 · Published 2026-07-17 · By Tam Bang, Hoang H. Nguyen, Lei Cheng, Lihao Guo, Siyang Cao, Hussam Abubakr et al.

TL;DR

This paper addresses the challenge of reliable roadside perception of vulnerable road users (VRUs) like pedestrians, cyclists, scooterists, and wheelchair users in complex urban intersections under adverse conditions such as occlusion, variable lighting, and weather. Existing multi-sensor fusion solutions rely on cloud or server-grade infrastructure, limiting real-world edge deployments which have strict latency and computational constraints. The authors propose CLIFE, a fully on-device, edge-native Camera–LiDAR fusion framework that integrates targetless, online calibration and lightweight late-fusion tracking on a single embedded NVIDIA Jetson device without cloud offloading. CLIFE dynamically refines calibration on demand and performs multi-sensor fusion with O(N log N) per-frame complexity, achieving a real-time fusion rate of 53.2 FPS on the Jetson AGX Thor.

The authors deploy CLIFE across 12 live intersections in Chattanooga, Tennessee and provide a detailed evaluation at a representative intersection. Their experiments show that the fusion system significantly extends perceptual range and robustness compared to single-sensor baselines under a variety of real-world lighting and weather conditions (sunny, cloudy, light rain, nighttime glare, heavy rain). The end-to-end fusion approach improves tracking accuracy and identity consistency compared to camera-only or LiDAR-only systems, especially when sensors individually degrade. The offline calibration module enables targetless, automated online calibration to recalibrate for mount drift or environmental perturbations without manual intervention or service interruptions. Overall, CLIFE provides a scalable, modular foundation for practical roadside VRU perception under edge deployment constraints.

Key findings

• The proposed late fusion core runs at 53.2 FPS on a Jetson AGX Thor embedded edge device.
• CLIFE improved tracking accuracy: MOTA increased from 67.7% (camera-only) and 76.4% (LiDAR-only) to 78.6% under sunny conditions.
• Identity F1 score (IDF1) rose from 73.6% (camera-only) and 84.3% (LiDAR-only) to 86.0% for fusion under sunny weather.
• Targetless calibration on rectified images converged to an average Euclidean distance (AED) of 16.02 pixels after approx. 30 minutes of on-site calibration.
• The framework supports up to five parallel camera-LiDAR fusion workers on a single Jetson AGX Thor at ~10 FPS per stream, sufficient for two full intersections plus one extra camera-LiDAR pair.
• YOLOv11-m with FP16 precision at 1280×1280 input achieved a detection [email protected]:0.95 of 0.7786 and 39.4 FPS on camera perception dataset of 25,270 frames.
• LiDAR perception done by the BlueCity API runs at 10 Hz; fully on-device PointPillars + AB3DMOT model achieves 47.6 FPS for LiDAR detections.
• Fusion performs best when combining camera semantic richness with LiDAR 3D localization, especially improving under occlusion, low light, and rain scenarios.

Threat model

The system assumes natural environmental variation and sensor imperfections as the main challenges, with an adversary implicitly represented by occlusions, sensor noise, and mount drift. It does not consider malicious attackers capable of spoofing, tampering with, or physically obstructing sensors. The design assumes honest-but-faulty sensors and aims to robustly fuse camera and LiDAR perception under edge compute and latency constraints.

Methodology — deep read

1. Threat Model & Assumptions: The system targets roadside VRU perception against natural environmental and traffic challenges, assuming no adversarial spoofing or attacks. The sensors include cameras and LiDAR mounted on roadside infrastructure, with real-time edge processing constraints. The adversary model is implicit: the system must tolerate occlusions, lighting and weather variation, sensor misalignment due to mount drift, but does not explicitly consider attacks on sensors themselves.

2. Data: Two custom datasets were curated from Chattanooga intersections: a camera perception dataset of 25,270 frames with VRU annotations from scripted scenarios for training and evaluation; and a synchronized camera-LiDAR dataset of 9,000 time-aligned frame pairs under varied weather (sunny, cloudy, light rain) used for end-to-end evaluation. Annotation involved auto-labeling and manual refinement for the camera dataset, and manual correction from BlueCity LiDAR outputs. Additional qualitative data under heavy rain and nighttime glare was collected but not labeled.

3. Architecture / Algorithm: CLIFE consists of two main phases. Phase 1 is a targetless online camera–LiDAR calibration that estimates a ground-plane homography matrix H by matching detections across modalities using spatial, appearance, and semantic cues, refined iteratively to minimize reprojection error. This calibration is triggered on demand to correct mount drift or misalignment without manual targets.

Phase 2 is a late-fusion pipeline that projects LiDAR detections into image space using the homography, then performs category-specific nearest-neighbor matching via KD-tree radius search and greedy one-to-one assignment, fusing matched detections by combining camera semantic class/confidence with LiDAR 3D position and velocity. Unmatched detections are passed through as single-sensor hypotheses. A multi-object tracker maintains persistent fused tracks, handling temporary single sensor misses via miss counters and track termination rules, preserving consistent IDs.

The camera perception module uses YOLOv11-m at 1280×1280 resolution fine-tuned on the custom dataset with ByteTrack for online frame-to-frame tracking. The LiDAR perception uses BlueCity API for merged multi-LiDAR point clouds at 10 Hz. Alternative on-device LiDAR processing uses PointPillars architecture with AB3DMOT tracking.

4. Training Regime: YOLOv11 was fine-tuned on 25,270 training frames split 80:10:10 for 1280×1280 input resolution, using FP16 precision for efficiency. The calibration model (CalibRefine) was fine-tuned on an intersection-specific calibration subset to adapt cross-modal matching to roadside sensor geometry.

5. Evaluation Protocol: Detection evaluated with standard metrics: precision, recall, [email protected] and [email protected]:0.95. Tracking evaluated by MOTA and IDF1 scores across environmental conditions (sunny, cloudy, light rain). Calibration accuracy evaluated by average pixel reprojection error (AED). Baselines include camera-only and LiDAR-only pipelines, compared with full fusion. Real-time throughput measured on embedded hardware. Qualitative analyses on hard scenarios (occlusion, rain, glare).

6. Reproducibility: The datasets are custom-collected and not publicly released. BlueCity LiDAR API backend is proprietary, but an open-source on-device LiDAR pipeline variant is evaluated. Model checkpoints and code release status are not specified, limiting reproducibility. The paper details sufficient architecture and deployment parameters for system replication but lacks open-source artifacts.

Example Walkthrough: For each synchronized camera frame and LiDAR sweep (both 10 Hz), camera detections produce 2D object bounding boxes with class confidence; LiDAR detections yield 3D bounding boxes. The online targetless calibration estimates updated homography H projecting LiDAR ground-plane points into the camera frame. Using H, LiDAR detections are projected and associated per object class with camera detections via nearest-neighbor search within a radius. Fused detections combine attributes, and unmatched detections pass through single-modality pipelines. Tracks are updated in a multi-frame buffer, maintaining consistent IDs despite misses. The entire pipeline runs onboard a Jetson AGX Thor edge device to enable real-time intersection-scale VRU perception with minimal latency and no cloud reliance.

Technical innovations

• Integration of an on-demand, targetless online camera–LiDAR calibration module that estimates ground-plane homography for continuous extrinsic alignment without manual targets.
• A lightweight, computationally efficient O(N log N) late fusion pipeline combining high-level camera and LiDAR detections instead of raw data, enabling fast multi-sensor association on embedded devices.
• Modular edge-native architecture that fully operates on a single embedded Jetson device without cloud offloading, maintaining real-time performance at 53.2 FPS fusion throughput.
• Demonstration of scalable deployment to multiple intersections with multi-stream fusion running in parallel, validated in a live urban environment with 12 deployed intersections.

Datasets

• Camera Perception Dataset — 25,270 frames — custom collected at Chattanooga intersections with scripted VRU scenarios, manually refined annotations
• Synchronized Camera–LiDAR Dataset — 9,000 frame pairs — custom collected under diverse sunny, cloudy, and light rain conditions at Chattanooga intersections

Baselines vs proposed

• Camera-only baseline: MOTA = 67.7%, IDF1 = 73.6% (sunny) vs Fusion: MOTA = 78.6%, IDF1 = 86.0%
• LiDAR-only baseline: MOTA = 76.4%, IDF1 = 84.3% (sunny) vs Fusion: MOTA = 78.6%, IDF1 = 86.0%
• Camera-only baseline: MOTA = 63.2%, IDF1 = 68.8% (cloudy) vs Fusion: MOTA = 65.4%, IDF1 = 71.0%
• LiDAR-only baseline: MOTA = 60.1%, IDF1 = 66.9% (cloudy) vs Fusion: MOTA = 65.4%, IDF1 = 71.0%
• Camera-only baseline: MOTA = 59.5%, IDF1 = 66.4% (light rain) vs Fusion: MOTA = 65.8%, IDF1 = 71.2%
• LiDAR-only baseline: MOTA = 70.1%, IDF1 = 78.3% (light rain) vs Fusion: MOTA = 65.8%, IDF1 = 71.2%
• YOLOv11-m FP16 at 1280×1280: [email protected]:0.95 = 0.7786 with 39.4 FPS on Jetson AGX Thor camera perception dataset

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.16154.

Fig 1: Roadside VRU perception challenges: (a) camera

Fig 2: Overview of the CLIFE architecture. The framework

Fig 3: illustrates the resulting alignment, where LiDAR

Fig 4: Street-view of the sensor setup.

Fig 5: Sample images from the multi-sensor VRU dataset.

Fig 6: LiDAR ground-plane projections overlaid on the orig-

Fig 7: Qualitative results demonstrating the robustness of our fusion method under challenging real-world scenarios. LiDAR-

Limitations

• Performance remains sensitive to extrinsic calibration accuracy despite targetless online refinement; misalignment can degrade fusion quality.
• Late-fusion approach propagates upstream detection errors directly, limiting robustness if individual sensors fail or produce false positives.
• Reliance on proprietary BlueCity LiDAR API backend constrains full end-to-end optimization and scalability with alternative open-source LiDAR processing.
• More extreme weather conditions and nighttime scenarios were only qualitatively evaluated due to annotation challenges limiting quantitative validation.
• The custom datasets and calibration models are not released publicly, limiting reproducibility and benchmarking against other fusion methods.
• The approach does not evaluate adversarial or spoofing attacks on sensors, focusing instead on natural environmental challenges.

Open questions / follow-ons

• How to reduce calibration sensitivity and improve robustness under extreme occlusions and aggressive sensor misalignment?
• Whether tighter end-to-end fusion architectures (e.g., intermediate or early fusion) can be adapted to edge constraints for further gains?
• How to incorporate cooperative V2X communication and additional sensing modalities to enhance robustness in complex urban intersections?
• How to extend quantitative evaluation and annotation techniques to support heavy rain, nighttime, and other adverse conditions reliably?

Why it matters for bot defense

Bot-defense and CAPTCHA practitioners focused on real-time systems can draw valuable insights from CLIFE’s design of a scalable, low-latency multi-sensor fusion pipeline optimized for edge deployment. The targetless online calibration approach exemplifies adaptive maintenance of sensor alignment without human intervention, which parallels the need for reliable, self-correcting detection in adversarial real-world settings. The late-fusion methodology demonstrates a practical balance between accuracy and computational efficiency useful for resource-constrained security deployments requiring multi-modal data fusion.

Although CLIFE targets roadside VRU perception, the architecture’s modularity and optimization for embedded devices provide a good blueprint for deploying robust perception or detection modules where latency, power, and reliability are critical. The demonstrated scalability and ability to handle diverse environmental conditions encourage further exploration of multi-sensor fusion for bot detection systems that combine visual, behavioral, or environmental cues under real-world deployment constraints.

Cite

@article{arxiv2607_16154,
  title={ CLIFE: Camera-LiDAR Fusion Framework for Edge-Deployable Roadside VRU Perception },
  author={ Tam Bang and Hoang H. Nguyen and Lei Cheng and Lihao Guo and Siyang Cao and Hussam Abubakr and Tianya Zhang and Austin Harris and Mina Sartipi },
  journal={arXiv preprint arXiv:2607.16154},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.16154}
}

Read the full paper

• arXiv:2607.16154 (abstract)
• arXiv:2607.16154 (PDF)