Bridging Performance and Generalization in Reinforcement Learning for Agile Flight

Source: arXiv:2606.27348 · Published 2026-06-25 · By Jonathan Green, Jiaxu Xing, Nico Messikommer, Angel Romero, Davide Scaramuzza

TL;DR

This paper addresses the critical limitation of existing reinforcement learning (RL) methods for autonomous drone racing: poor zero-shot generalization (ZSG) to unseen racetracks. Although prior RL controllers can achieve human-level or better performance on fixed tracks, they fail catastrophically when faced with novel environments—often crashing on the very first attempt. This gap arises from the complexities of agile flight, including high-dimensional task variation and a strict coupling of safety and speed. Previous approaches improving generalization did so by sacrificing flight speed and performance. The authors present a novel framework combining adaptive task switching based on learning progress metrics with a physically informed procedural racetrack generator that produces feasible and diverse tasks. Their training pipeline runs multiple racetracks in parallel, replacing tasks that no longer provide learning progress with newly generated ones.

The resulting generalist policy demonstrates a 7.4× improvement in zero-shot generalization over state-of-the-art methods while maintaining competitive racing speeds—about 37.7% faster than prior generalist methods and only 14.5% slower than specialized single-track policies. This improvement strongly holds in both simulation and real-world drone deployments, including a challenging vision-based end-to-end control setup that relies on pixel inputs without explicit state estimation. The vision-based policy achieves 2.3× better ZSG than existing vision methods, representing the first demonstration of zero-shot generalization in vision-based agile drone racing. Detailed ablations reveal the critical roles of adaptive task switching, informed procedural track generation, and carefully tuned regularization. These results systematically bridge the longstanding trade-off between robustness/generalization and performance in high-speed agile flight for RL agents.

Key findings

• Generalist agent (state-based) achieves a maximum Performance-Weighted Success Score (Spw) of 0.1652 vs. 0.0222 for Environment as Policy (EaP) and 0.000325 for Single-Task (ST) PPO, a 7.4× improvement in generalization (Fig. 3 Left).
• State-based generalist policy completes unseen core tracks with an average 37.73% faster lap times than EaP and only 14.52% slower than ST specialized policies (Table 1).
• Vision-based generalist reaches Spw = 0.05219, 2.3× better than state-based EaP despite more difficult pixel input observation.
• Real-world deployment on four core racetracks yields 100% success rate and an average sim-to-real lap time increase of only 6.575%.
• Adaptive task switching based on reward flatness detection (Spearman or Kalman) improves generalization by ~10.3% over no switching; random task switching degrades performance by 2.96%.
• Informed spline-based racetrack generation doubles zero-shot generalization compared to uniform random track sampling (Spw 0.1652 vs 0.08069).
• Careful L2 regularization improves generalization; optimal weight decay at 1.0×10^-5, with stronger regularization destabilizing training.
• Single-task trained policies memorize specific trajectories, failing to adapt or generalize even to minor gate position changes (Fig 6).

Threat model

The adversary is the unknown racetrack, representing an unseen task sampled from a continuous distribution of possible racetrack layouts. The agent does not receive any adaptation or fine-tuning on these new tracks (zero-shot). It must generalize robustly without prior knowledge of the layout, relying solely on training experience over diverse racetracks. No malicious entities or adversarial perturbations are considered.

Methodology — deep read

1. Threat Model and Assumptions: The adversary in this context is the deployment environment—unseen racetrack layouts unknown during training. The agent must generalize zero-shot without fine-tuning or adaptation on these new tasks. No malicious attackers are modeled; the challenge is natural distribution shift to novel task parameters. The environment parameters (racetrack layouts) are constant per episode, but vary across tasks.

2. Data: Training tasks are racetracks defined by gate positions and orientations. The training set T_train is a subset of the overall track distribution T. To generate training tasks, the authors propose a procedural spline-based track generation method that enforces physical feasibility (smoothness, connectivity) and geometric diversity. This contrasts with prior uniform random task sampling, which produced many infeasible or uninformative tracks. Evaluation tasks T_test include four standard human-designed tracks (Figure8, BigS, Kidney, SplitS) held out from training. Observations include both full system states (rotation matrices, velocities, relative gate positions) and vision-based observations (84×84 images encoding gate brightness to indicate order).

3. Architecture and Algorithm: The policy uses Proximal Policy Optimization (PPO) with a Multi-Layer Perceptron (MLP) policy network of two 512-unit layers for state-based agents. Vision-based agents add a 3-layer CNN encoder producing a 128-dimensional image embedding concatenated with partial state info. The actor receives previous actions, gate-relative positions or embeddings, and velocity info. The critic always receives state info. The authors introduce two novel training mechanisms: (a) Adaptive Task Switching detects when training on a task plateaus by analyzing recent reward trends per task, triggering replacement with new tasks. Two flatness detection methods are used: Spearman rank correlation and a Kalman filter with a local linear trend model. (b) Informed Task Generation uses spline-based track generation constrained by smoothness, physical constraints, and diversity criteria, ensuring tasks are feasible and diverse to provide meaningful learning signals.

4. Training Regime: The training runs large-scale parallel rollouts across multiple tasks simultaneously, periodically replacing plateaued tasks with newly generated tracks. PPO optimization uses a tunable L2 regularization (weight decay) parameter; the best value is 1.0×10^-5. Vision-based training requires an image encoding with gate brightness to indicate ordering. Training proceeds for approximately 20,000 iterations, with thousands of task switches throughout.

5. Evaluation Protocol: Performance is measured by a Performance-Weighted Success Score (Spw), combining success rate (completion without crashing) and lap time (speed) averaged over all unseen evaluation tracks. Comparative baselines include Environment as Policy (EaP), a prior multi-task approach, and single-task PPO policies trained on individual tracks. Metrics are reported on simulation and on real drone hardware, measuring sim-to-real gap. Ablations disable components (adaptive switching, informed generation, regularization) to assess their impact. Statistical confidence intervals and learning curves illustrate the significance of improvements.

6. Reproducibility: The paper does not mention public code release or open datasets but does provide algorithmic details and track generation procedures in appendices. The simulator is specialized to rapid track generation for parallel training. Data splits and task parameter distributions are described. The approach requires substantial compute due to parallelized multi-task training.

Concrete Example: The method concurrently trains a single policy across a dynamically maintained set of racetracks. When reward plateau is detected via Spearman rank coefficient on a specific track, that track is replaced with a procedurally generated spline-based feasible track differing in geometry. This forces the policy to continually learn on novel challenging tasks rather than overfitting. The final policy thus learns generalizable flight strategies adaptable to new racetrack geometries and high-speed control without relying on memorized trajectories, validated by robust zero-shot success and fast lap times on unseen tracks in simulation and physical flight tests.

Technical innovations

• Adaptive task switching mechanism utilizing online reward flatness detection (Spearman rank and Kalman filter) to dynamically replace plateaued tasks during multi-task RL training.
• Physically informed procedural racetrack generation using B-Splines with feasibility and diversity constraints to produce challenging, realistic training tasks.
• Demonstration of zero-shot generalization in vision-based high-speed drone racing policies using specialized gate brightness embeddings combined with compact CNN image encoding.
• Large-scale parallel multi-task reinforcement learning framework that avoids catastrophic forgetting by joint updates across continually updated task sets.

Datasets

• Training racetrack tasks generated procedurally via spline-based task generator — unquantified number, continuous parameterized distribution — simulator-internal
• Evaluation racetracks: Figure8, BigS, Kidney, SplitS — 4 main human-designed tracks held out from training — sourced from prior work [3]

Baselines vs proposed

• Environment as Policy (EaP): Spw = 0.0222 vs. Proposed generalist: Spw = 0.1652
• Single-Task PPO: Spw = 0.000325 vs. Proposed generalist: Spw = 0.1652
• State-based generalist lap time: 37.73% faster than EaP on core tracks
• State-based generalist lap time: 14.52% slower than specialized Single-Task PPO
• Vision-based generalist: Spw = 0.05219 vs. state-based EaP: ~0.022 (2.3× improvement)
• Single-task (ST) policies crash on unseen track modifications causing zero success rate; generalist policies maintain 60-70% success without adaptation

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.27348.

Fig 1: Our policy achieves zero-shot generalization on diverse unseen racetracks while maintain-

Fig 2: Overview of the training pipeline and model architecture. For the state-based agent, the

Fig 3 (page 1).

Fig 4 (page 1).

Fig 8: 2.940±0.4376

Fig 3: Left: Comparison of the generalization performance of this and other works. “SB” de-

Fig 7 (page 3).

Fig 8 (page 3).

Limitations

• Extrapolation to out-of-distribution track families, dynamic moving gates, and obstacle-filled tracks remains untested.
• Vision-based policies require partial explicit state inputs (rotation and velocity) for stability, limiting full end-to-end autonomy from pixels alone.
• Current approach relies on computationally expensive large-scale parallel training and fast simulation, limiting accessibility.
• No formal verification or provably safe fallback controllers included, so safety under real-world sensor failures or extreme conditions is uncertain.
• Sim-to-real gap, while small (~6.5%), could increase under lighting variations, motion blur, or degraded perception.
• Catastrophic forgetting is mitigated but not eliminated; long-term continual learning scenarios not evaluated.

Open questions / follow-ons

• How well does the framework generalize to dynamic environments with moving obstacles or gates?
• Can fully vision-based policies operate robustly without any explicit state inputs like velocity or rotation?
• What scalable methods can improve sample efficiency to reduce training compute demands?
• How can safety guarantees or fallback mechanisms be integrated to enable trusted deployment in safety-critical environments?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this paper exemplifies the challenge of balancing performance with robust generalization under distribution shifts—a critical concern when designing systems to detect or resist automated agents across unpredictable input variations. The adaptive task-switching mechanism provides a rigorous approach to continuously challenge learning agents with novel, feasible variations, preventing overfitting and promoting resilience. The emphasis on procedural generation of training tasks that respect physical constraints ensures that training distributions closely approximate realistic deployment conditions, improving real-world reliability.

In CAPTCHA or bot-detection settings, insights from this work suggest that deploying models trained on diverse, carefully constructed task distributions combined with meta-learning of when to introduce new challenges can improve zero-shot robustness against novel bot strategies or input perturbations. The demonstrated success of using learning progress as a trigger for curriculum updates may inspire similar progress-based challenge adaptation to maintain defender efficacy. However, the computational costs and environment-specific tuning should be carefully considered in operational settings.

Cite

@article{arxiv2606_27348,
  title={ Bridging Performance and Generalization in Reinforcement Learning for Agile Flight },
  author={ Jonathan Green and Jiaxu Xing and Nico Messikommer and Angel Romero and Davide Scaramuzza },
  journal={arXiv preprint arXiv:2606.27348},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.27348}
}

Read the full paper

• arXiv:2606.27348 (abstract)
• arXiv:2606.27348 (PDF)