Real-Time Visual Intelligence on Low-Cost UAVs: A Modular Approach for Tracking, Scanning, and Navigation
Source: arXiv:2607.02298 · Published 2026-07-02 · By Andrei-Marian Ungureanu, Stelian Spînu
TL;DR
This paper addresses the challenge of deploying advanced real-time visual intelligence on low-cost, resource-constrained drones. The authors present a modular AI system implemented on the DJI Tello platform, which integrates three core functionalities: face detection (using YuNet), facial recognition (using FaceNet), and monocular depth estimation (using DepthAnythingV2). The system runs on a Python-based backend on an external laptop/server that controls the drone via the DJITelloPy SDK and communicates through low-latency UDP. A web interface provides user control and real-time telemetry.
The contributions include autonomous person tracking with a FollowMe module using face centroid and bounding box area for spatial control via PID loops; a SkyScan module that performs indoor room scanning with monocular depth for navigation and integrates facial recognition with a cloud-hosted database; and a LineTracker module that employs a custom mirror mount and a virtual sensor matrix for visual-based path following without GPS. Experimental results show robust tracking accuracy (>92%), high face recognition accuracy (97.4%), and reliable line following on complex routes with smooth velocity adaptation. The modular system achieves low-latency real-time control while running entirely on accessible, low-cost hardware without onboard sensors or GPS. This validates the feasibility of building intelligent UAV assistants on consumer-grade platforms, providing a scalable open-source baseline for research in autonomous drones for surveillance, rescue, or logistics in GPS-denied environments.
Key findings
- FollowMe module achieved >92% tracking accuracy over 50+ trials, with average positional correction latency <300 ms and reacquisition after occlusion under 2 seconds in 89% of cases.
- SkyScan module's YuNet face detector succeeded in >95% of frames even in medium lighting and cluttered backgrounds.
- FaceNet recognition accuracy reached 97.4%, with no false positives on test sets over 20 individuals.
- LineTracker completed all 10 predefined paths with average lateral deviation <15 pixels and no off-path failures, dynamically adjusting speed around turns.
- Integrated system latency: <100 ms for FollowMe, <150 ms for SkyScan, and <120 ms for LineTracker; CPU utilization remained below 65% on AMD Ryzen7 laptop.
- System runs on low-cost DJI Tello drone with no onboard depth sensors or GPS, using a 3D-printed mirror mount to enable downward path tracking.
- Temporal consistency filtering was applied in face tracking and recognition to improve robustness against false detections.
- Use of monocular depth estimation (DepthAnythingV2 ViT-B) allowed indoor navigation and scanning up to 20 meters without additional hardware.
Threat model
The adversary consists primarily of environmental challenges such as partial occlusions, lighting variability, dynamic obstacles, and motion unpredictability that may cause detection failures or tracking loss. No adversarial attacks against the AI models or communications are considered. The system assumes trusted and uninterrupted command and telemetry channels and operates in GPS-denied but indoor environments where external sensing infrastructure is unavailable.
Methodology — deep read
The paper defines a threat model focusing on achieving robust autonomous operation in GPS-denied, indoor or cluttered environments using low-cost drones, with the adversary being environmental challenges such as occlusions, lighting changes, and motion rather than active attackers.
Data sources include video streams from the DJI Tello's monocular RGB camera. Face detection was performed using YuNet, a lightweight and efficient face detector capable of running under low computational resources. Facial recognition relied on FaceNet embeddings matched against a cloud-stored Firebase Realtime Database with known identities. Depth estimation was based on DepthAnythingV2, a Vision Transformer (ViT-B) monocular depth estimator designed for diverse real-world indoor scenes without requiring stereo setups.
The FollowMe module uses real-time face detection outputs (bounding box centroid and area) to control the drone's lateral and forward/backward movement via PID controllers. The drone maintains the face in the center of the frame and adjusts distance based on bounding box size as a proxy for depth.
The SkyScan module performs autonomous scanning of indoor spaces by combining monocular depth maps from DepthAnythingV2 with facial recognition results to identify humans during the sweep. The module leverages depth to dynamically adjust yaw angles and estimate room width, enabling scan coverage without physical depth sensors.
The LineTracker module enables the drone to follow ground lines visually by reflecting the downward floor view into the fixed forward camera using a custom 45° mirror mount. Frames are converted to HSV color space, thresholded to isolate lines, and segmented into a simulated 1×3 virtual sensor matrix inspired by prior work. The distribution of line pixels across these zones drives proportional yaw corrections, and adaptive speed control slows the drone on tight curves for stability.
All modules run on a Python-based multi-threaded backend server on a standard laptop (AMD Ryzen7 CPU), taking the drone video feed, processing inferences, and sending periodic flight commands over UDP through the DJITelloPy SDK. Temporal filtering enforces consistency by requiring repeated detections before action.
Training involved using pretrained models (YuNet, FaceNet, DepthAnythingV2) optimized for CPU-based embedded inference but no new deep learning training was performed by the authors. Hyperparameters for PID controllers and recognition thresholds were experimentally chosen.
Evaluation was conducted in controlled indoor scenarios with variable lighting, obstacles, and human motion. Metrics included tracking accuracy, recognition accuracy, path-following deviation, latency, CPU utilization, and reacquisition times. Modules were evaluated individually and in integrated operation to assess concurrency and resource demands.
Code is publicly available on GitHub, but the dataset (video streams and controlled environments) appear not to be fully public. The reliance on offboard processing limits portability, and no formal cross-validation or adversarial testing beyond induced occlusions is reported.
A concrete example: the FollowMe module detects a face bounding box, computes centroid offset from frame center, derives PID adjustments for yaw and lateral movements, estimates distance from bounding box area, and adjusts forward/backward throttle commands accordingly, continuously updating these controls at about 10-15 Hz ensuring smooth person following behavior.
Technical innovations
- Modular integration of face detection, facial recognition, and monocular depth estimation on a low-cost UAV platform using only its RGB camera and no additional specialized sensors.
- Use of bounding box area from face detection as a proxy depth cue to enable distance regulation in person-following without dedicated range sensors.
- Custom 3D-printed mirror mount to reflect downward floor view into fixed front-facing drone camera enabling visual path following with virtual sensor arrays.
- Temporal consistency filtering across multiple frames to reduce false positives in both face tracking and recognition modules, enhancing operational robustness.
- Adaptive speed control in vision-based line following based on real-time analysis of line curvature derived from pixel density distributions in virtual sensor zones.
Datasets
- Internal indoor drone video datasets for face detection, recognition, and navigation tasks — approximate size not specified — collected by authors in controlled environments
Baselines vs proposed
- FaceNet vs VGGFace and GhostFaceNets: Accuracy = 97.4% (FaceNet) vs lower for others as per Fig. 4
- YuNet face detector vs traditional detectors (Haar Cascades, MTCNN): millisecond-level inference time for YuNet enabling real-time operation vs slower baseline detectors
- FollowMe tracking accuracy = 92% with <300 ms correction latency (no direct numerical baseline but comparable literature suggests competitive performance)
- LineTracker deviation <15 pixels vs prior matrix sensor designs with reportedly worse angle anticipation [4], illustrating improved predictive path following
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.02298.
Fig 1: Face outside the center zone
Fig 2: Face area
Fig 3: Figure 3. Performance of face recognition models on masked faces
Fig 4: Accuracy of face recognition models
Fig 5: Physical setup used for path-following
Limitations
- All inference and control logic runs offboard on a laptop; no onboard AI computation on the drone itself limits portability and autonomy.
- Depth estimation relies solely on monocular vision without stereo or active sensors, which may reduce accuracy in complex or cluttered environments.
- Facial recognition is limited to single-face tracking; no multi-face or crowd-scene handling explored.
- Testing performed in controlled indoor environments; performance under outdoor lighting, weather, or adversarial visual conditions remains untested.
- PID controller parameters were empirically tuned; no formal stability or robustness proofs provided.
- System latency and CPU load measured on a standard laptop; feasibility on embedded AI hardware platforms is suggested but not demonstrated.
Open questions / follow-ons
- Can critical AI inference and flight control be migrated onboard the UAV using embedded AI accelerators (e.g., Jetson Nano, Coral TPU) to enable fully autonomous flight without external servers?
- Would integrating active depth sensors like stereo cameras or time-of-flight units improve depth estimation accuracy and obstacle avoidance in cluttered spaces?
- How can the system be extended to handle multi-person tracking, multi-face recognition, and dynamic crowd scenarios without degradation in performance?
- What benefits would the incorporation of sensor fusion, visual SLAM, and trajectory prediction techniques bring to robustness and navigation in complex, unknown indoor environments?
Why it matters for bot defense
This paper provides valuable insights for bot-defense engineers interested in real-time visual intelligence under resource constraints typical to embedded or UAV platforms. The modular architecture and use of lightweight, open-source AI models illustrate practical pathways to achieving robust person tracking, facial recognition, and navigation solely from video feeds without expensive sensors or GPS.
For CAPTCHA and bot mitigation, the FollowMe and SkyScan modules showcase how facial detection and ID recognition can be integrated within low-latency pipelines suitable for embedded applications, potentially informing defenses that must operate on constrained devices or in challenging environments. The temporal filtering strategies and combination of spatial cues for depth estimation may inspire techniques to improve robustness against spoofing or adversarial obfuscation in visual authentication. Additionally, the line-based navigation component exemplifies minimal-infrastructure guidance that could inspire covert bot detection via environmental cues.
Overall, this work underscores the importance of modular, flexible AI system design combined with tailored hardware adaptations that are pertinent when developing bot-detection capabilities or CAPTCHA systems embedded in lightweight devices.
Cite
@article{arxiv2607_02298,
title={ Real-Time Visual Intelligence on Low-Cost UAVs: A Modular Approach for Tracking, Scanning, and Navigation },
author={ Andrei-Marian Ungureanu and Stelian Spînu },
journal={arXiv preprint arXiv:2607.02298},
year={ 2026 },
url={https://arxiv.org/abs/2607.02298}
}