Calibration-Free 3D Multi-Camera People Tracking for Indoor Environment

Source: arXiv:2607.22731 · Published 2026-07-22 · By Ponleur Veng, Dominique Vaufreydaz, Phutphalla Kong

TL;DR

This paper addresses the challenge of Multi-Camera People Tracking (MCPT) in indoor environments without relying on manual camera calibration, which is traditionally required to unify 2D detections into a 3D world coordinate system. Manual calibration is labor-intensive and a bottleneck for scalable dataset generation, especially from unconstrained video archives. The authors propose a novel, end-to-end calibration-free framework that infers relative camera poses and scene geometry directly from the visual data using transformer-based foundation models. Integrating state-of-the-art components—YOLOX for detection, BoT-SORT for robust tracking, OsNet for omni-scale appearance embeddings, HRNet for pose estimation, and the Visual Geometry Grounded Transformer (VGGT) for geometric reconstruction—they replace classic ground-plane homography with a pose-guided 3D lifting strategy leveraging head keypoints. This improves robustness in crowded, occluded indoor scenes.

Evaluated on the AI City Challenge 2024 benchmark, a large-scale dataset with over 2,400 identities and 95 cameras, the system achieves a competitive HOTA score of 53.13% without access to any ground-truth calibration matrices. This performance surpasses several calibrated state-of-the-art baselines, confirming the efficacy of calibration-free, vision-only geometry inference for practical 3D multi-camera tracking. Detailed ablations show that combining appearance and geometric cues yields the strongest results, while purely appearance-based or geometric-only models perform worse. The work establishes a strong baseline for scalable 3D people tracking systems that require minimal manual setup, crucial for deployment in real-world unconstrained indoor scenarios.

Key findings

• The proposed calibration-free MCPT model achieves a HOTA score of 53.13% on the AI City Challenge 2024 dataset, exceeding the calibrated ARV baseline's 51.05% (Table 3).
• Using only appearance embeddings for identity association yields high association accuracy (93.12%) but low detection accuracy (30.55%), highlighting spatial inconsistency under occlusion (Table 2).
• Using only geometric constraints results in better spatial detection accuracy (33.71%) but lower association accuracy (79.53%), indicating that geometry alone lacks discriminative power.
• Balanced fusion of appearance and geometry costs with weights Wapp=0.5 and Wgeo=0.5 achieves optimal performance: HOTA 53.13%, DetA 38.39%, AssA 80.16% (Table 2).
• The pose-guided 3D lifting approach using head keypoints eliminates dependency on ground-plane homography and improves robustness in indoor occluded scenes.
• VGGT transformer can recover dense 3D point clouds and relative camera poses from uncalibrated frames, enabling calibration-free multi-view fusion.
• Strict velocity gating (transitions penalized if speed > 4.0 m/s) improves physical plausibility in identity association.
• Localization accuracy (LocA) remains around 62.18% across all configurations, mainly limited by YOLOX detector quality rather than association.

Threat model

The adversary is an indoor scene with multiple uncalibrated cameras capturing video of people, without knowledge or access to ground-truth camera intrinsic or extrinsic parameters. The adversary cannot directly manipulate the visual scene inputs but can present heavy occlusion, dense crowding, and overlapping fields of view to challenge tracking consistency. The system assumes static synchronized cameras and does not consider adversarial attacks such as spoofing or injection.

Methodology — deep read

The authors propose a fully calibration-free 3D multi-camera tracking pipeline designed for indoor environments with dense occlusions and no available camera calibration data. The threat model implicitly considers a scenario where camera intrinsics, extrinsics, and ground planes are unknown, and the system must rely solely on visual input from synchronized multiple static cameras.

The data consists of synchronized multi-view video frames from the AI City Challenge 2024 Track 1 dataset, which includes 90 scenes, 953 cameras, 2,491 identities, and over 100 million bounding boxes at 1920x1080 resolution and 30 FPS, totaling 212 hours of video. The dataset includes dense 3D ground-truth annotations used for evaluation. The authors use standard splits as defined by the challenge and process raw video frames without explicit calibration parameters.

The pipeline integrates several modular components: (1) YOLOX, an anchor-free detector, produces bounding boxes with confidence scores per frame; (2) BoT-SORT performs robust 2D tracking, generating tracklets and using a Kalman filter state vector for bounding box position and velocity prediction; (3) OsNet extracts omni-scale appearance embeddings averaged over tracklet frames to facilitate re-identification; (4) HRNet, trained on CrowdPose and implemented via MMPose, estimates anatomical head keypoints, which are preferable in occluded indoor scenarios; (5) VGGT, a transformer-based visual geometry grounded model, processes pairs of uncalibrated frames to infer relative camera rotations, translations, and dense 3D reconstructed point clouds without any explicit calibration.

A key novel step is the pose-guided 3D lifting that projects detected 2D head keypoints onto the reconstructed 3D manifold generated by VGGT, circumventing classical ground-plane homography assumptions. This lifting facilitates real-world 3D localization under heavy occlusion.

Global trajectory association across cameras is formulated as a hierarchical agglomerative clustering (HAC) problem that combines a joint cost function with weighted appearance (cosine distance of OsNet features) and geometric components (normalized Euclidean distance of 3D head points). A strict velocity gating rule penalizes physical impossibility by heavily weighting transitions requiring speeds over 4.0 m/s. The clustering threshold is set to tau=0.5 for final identity output.

The model training details for individual components like YOLOX, OsNet, and HRNet follow prior established protocols and are not retrained end-to-end. VGGT weights are from pretrained foundation models [22]. The evaluation uses the Higher Order Tracking Accuracy (HOTA) metric computed in world coordinates, incorporating detection accuracy (DetA), association accuracy (AssA), and localization accuracy (LocA). Ablations explore varying weightings of appearance and geometric costs, cross-validating on a validation subset from the challenge.

Reproducibility notes: VGGT and other backbone models used are publicly available as foundation models, while the code integrating these into the pipeline is not explicitly stated as released. The AI City Challenge dataset is publicly accessible under controlled terms, enabling replication of evaluation. Overall, the paper demonstrates an end-to-end example of multi-camera videos processed through detection, pose estimation, transformer inference for scene geometry, and clustering to output calibrated-free 3D trajectories.

Technical innovations

• Application of a Visual Geometry Grounded Transformer (VGGT) for dense 3D reconstruction and relative camera pose estimation without explicit calibration.
• Pose-guided 3D lifting strategy that projects head keypoints onto the inferred 3D manifold, eliminating reliance on fragile ground-plane homography.
• Joint hierarchical agglomerative clustering for global identity association combining appearance and geometry costs with strict velocity gating.
• Integration of strong anchor-free detection (YOLOX), robust tracking (BoT-SORT), omni-scale ReID embeddings (OsNet), and anatomical pose estimation (HRNet) into a unified calibration-free 3D tracking framework.

Datasets

• AI City Challenge 2024 Track 1 — 212 hours of video, 953 cameras, 2,491 identities, >100 million bounding boxes — public benchmark

Baselines vs proposed

• Team Yachiyo [28]: HOTA = 71.94% vs proposed: 53.13%
• SJTU-Lenovo [26]: HOTA = 67.22% vs proposed: 53.13%
• Nota [11]: HOTA = 60.93% vs proposed: 53.13%
• Fraunhofer IOSB [19]: HOTA = 60.88% vs proposed: 53.13%
• UW-ETRI [27]: HOTA = 57.14% vs proposed: 53.13%
• ARV [3]: HOTA = 51.05% vs proposed: 53.13% (proposed outperforms ARV without calibration)

Limitations

• Absolute scale ambiguity persists since no external references are used; metric scale recovery depends on the inferred geometry.
• Depth estimation degrades in texture-poor regions, potentially reducing spatial accuracy in some scenes.
• The method currently does not match the top fully calibrated methods in HOTA and requires offline processing without online capabilities.
• No explicit adversarial robustness evaluation against malicious actors attempting to spoof geometry or appearance.
• Evaluation is limited to the AI City Challenge dataset; generalization to other environments or camera setups is unclear.
• Reproducibility is partially limited by lack of explicit public release of the full integration pipeline code.

Open questions / follow-ons

• How can adaptive scale recovery methods be developed to resolve absolute metric ambiguities inherent in calibration-free 3D reconstruction?
• What temporal smoothing or filtering strategies can improve 3D stability and reduce jitter in the reconstructed trajectories?
• Can the approach be extended to online, real-time tracking scenarios with moving or unsynchronized cameras?
• How well does the method generalize to outdoor or less structured environments with different lighting and scene complexities?

Why it matters for bot defense

This work is relevant to bot-defense and CAPTCHA practitioners interested in robust multi-camera human tracking without relying on burdensome manual calibration. Eliminating calibration lowers operational overhead for deploying 3D tracking infrastructures in indoor public spaces such as retail stores or transit hubs, where bot detection might rely on consistent person identity tracking across cameras. The pose-guided 3D lifting method also enhances robustness under severe occlusion, common in crowded environments relevant to bot detection. Integrating vision-only geometric inference offers a promising path to scalable, deployment-friendly multi-camera tracking systems that maintain tight cross-view identity association critical for behavioral analytics and anomaly detection. Nevertheless, practitioners should be aware of the current method’s offline nature, scale ambiguity, and limitations on absolute spatial accuracy when applying it to security or bot defense scenarios requiring real-time responses or metric distance measurements.

Cite

@article{arxiv2607_22731,
  title={ Calibration-Free 3D Multi-Camera People Tracking for Indoor Environment },
  author={ Ponleur Veng and Dominique Vaufreydaz and Phutphalla Kong },
  journal={arXiv preprint arXiv:2607.22731},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.22731}
}

Read the full paper

• arXiv:2607.22731 (abstract)
• arXiv:2607.22731 (PDF)