The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation

Source: arXiv:2607.24720 · Published 2026-07-27 · By Tianyi Men, Zhuoran Jin, Kang Liu, Jun Zhao

TL;DR

This paper targets the core challenge of systematically improving multi-turn long-horizon planning ability in foundation model agents. Unlike prior black-box training on Internet data, the authors design a fully controllable multi-turn environment that enables precise manipulation of task length, data quality, planning knowledge, and patterns. They study how long-horizon planning emerges and evolves across three key training stages: (1) acquisition during pre-training governed by data format, distribution, and quality; (2) shaping via reinforcement learning post-training with group relative policy optimization (GRPO) and on-policy distillation (OPD); and (3) integration from multiple teachers via multi-teacher on-policy distillation (MOPD).

Their experiments show that explicitly modeling a world model using chain-of-thought state transitions greatly enhances generalization on long-horizon tasks versus direct action prediction. Adding even a small fraction of long-horizon data during pre-training boosts compositional generalization beyond atomic skills alone. Suboptimal trajectories significantly degrade planning due to error compounding. At the post-training stage, OPD outperforms GRPO, exhibiting a broader effective region for improving planning under low data quality and longer horizons by providing more consistent update directions. However, distilling new, unseen multi-path planning knowledge from teachers with different underlying distributions can impair student models, causing partial forgetting. Multi-teacher OPD enables capability integration by converging to shared planning patterns across environments, facilitating cross-environment generalization and continual learning when patterns align, but causing interference when they conflict.

Overall, the paper offers a unified physics-inspired framework and extensive controlled experiments revealing how multi-turn long-horizon planning ability is acquired, shaped, and integrated across training stages. This framework and insights enable principled development and diagnosis of stronger agentic foundation models suited for complex sequential decision making.

Key findings

• World model internalization using chain-of-thought state transition modeling improves long-horizon generalization by +39.3% avg@8 and +32.9% pass@8 in middle and +45.8% avg@8 and +42.9% pass@8 in long-horizon tasks compared to direct action prediction (Fig 3).
• Adding a small amount (5%) of long-horizon data during pre-training significantly boosts compositional generalization on long-horizon tasks where atomic skills alone fail (Fig 3).
• Presence of suboptimal trajectories in training data severely impairs long-horizon planning ability, with drastic performance drops up to near 0% pass@8 due to error accumulation over long horizons (Section 4.3).
• OPD (on-policy distillation) post-training exhibits a broader and more robust effective region across horizon length and data quality dimensions than GRPO, due to more consistent gradient update directions (Fig 1,d,e).
• Attempting to distill unseen multi-path planning knowledge from teachers with different procedural knowledge can impair the student’s in-domain performance and world modeling, revealing limits of transfer when teacher and student knowledge conflict (Section 5.3).
• Multi-teacher on-policy distillation (MOPD) converges students to shared planning pattern distributions, enabling cross-environment generalization when patterns are compatible, continual learning when partially shared, and catastrophic forgetting when patterns conflict (Section 6).
• Direct action prediction is more sample efficient in early training but exhibits lower performance ceilings compared to planning with internalized world models (Fig 4).
• Sparse credit assignment in GRPO leads to noisy and less reliable gradient directions than OPD when shaping planning patterns on long-horizon, low-quality data (Section 5).

Threat model

The threat model is implicit rather than explicitly security-focused. The 'adversary' corresponds to the challenges posed by uncontrolled, noisy, suboptimal, or conflicting training data trajectories encountered during pre-training and post-training, which can degrade the agent’s long-horizon planning ability by propagating and amplifying errors. The study assumes no direct adversarial attacks in the classic sense but analyzes how data quality and distribution impact planning robustness and transfer.

Methodology — deep read

The authors study multi-turn long-horizon planning via a three-stage experimental scheme: acquisition during pre-training; shaping via post-training with RL/distillation; and integration through multi-teacher distillation.

1. Threat model & assumptions: The adversary is implicitly the opaque and uncontrollable environment/Internet data distributions that hinder understanding of long-horizon planning development. The authors assume full control over a synthetic multi-turn long-horizon planning gym environment but do not explicitly model adversarial attacks.

2. Data: They construct a controllable multi-turn environment with 3 domains (Fantasy Alchemy, Livestock Farming, Electronic Assembly), each containing hierarchical skill graphs. The task is to synthesize a target item by composing prerequisite skills encoded as AND/OR logic over item categories. The environment supports precise control over task length (short, middle, long steps), data quality (optimal/suboptimal trajectories), knowledge (planning patterns/skills), and distribution splits (pre-training, post-training, test).

3. Architecture/algorithm: The base model follows the Qwen2.5 architecture, trained from scratch with a custom byte-level BPE tokenizer for the synthetic corpus. Models differ in training objectives:

• Direct action prediction baseline predicts next action step.
• World modeling baseline uses chain-of-thought intermediate state transition prediction to internalize a world model, explicitly simulating environment dynamics during planning.
• Post-training uses Group Relative Policy Optimization (GRPO) RL or On-Policy Distillation (OPD) to refine planning patterns and knowledge.
• Multi-Teacher OPD (MOPD) distills from multiple teacher policies to integrate different planning capabilities.

4. Training regime: Pre-training uses 1.2 billion synthetic tokens with a mixture of optimal and suboptimal trajectories. Post-training involves reinforcement learning-like policy optimization over multi-turn trajectories, with care to analyze gradient consistency and update stability between GRPO and OPD. Multi-teacher distillation runs multiple teachers with different environment distributions for consolidation. Details on epochs, batch size, seeds are referenced in appendices but not explicitly enumerated in main text.

5. Evaluation protocol: Performance is evaluated on reserved test sets with unseen item combinations. Metrics include avg@8 (average success over 8 sampled trajectories) and pass@8 (tasks solved at least once in 8 runs), measured across short, middle, and long horizon tasks for 3 domains. Ablations compare with/without world model, adding long-horizon data, different proportions of suboptimal trajectories, and various post-training algorithms. Compatibility of planning patterns across environments is studied under multi-teacher setting. Statistical significance testing is not mentioned explicitly.

6. Reproducibility: Authors release code, models and datasets on GitHub and Huggingface. The environment, data generator, and synthetic corpora are publicly available for replication. Some hyperparameter specifics and detailed architectures are in appendices.

Example end-to-end flow: For a target item (e.g., cardigan), the agent initializes with starting materials. Using world model internalized in chain-of-thought, it simulates state transitions to plan intermediate steps (e.g., merge seeds to yarn, yarn to wool, wool and needle to sweater) recursively until target synthesis is achieved. The policy is distilled on-policy from a teacher generating optimal trajectories, optimized to minimize token-level KL divergence across multi-turn rollouts. Success is measured by final synthesis success over multiple sampled attempts at different horizon difficulty levels.

Technical innovations

• Introduction of a unified and fully controllable multi-turn long-horizon planning environment supporting precise control over task length, data quality, planning knowledge and patterns, enabling systematic study of long-horizon planning across stages.
• Explicit internalization of world models via chain-of-thought state transition modeling that significantly improves long-horizon generalization compared to direct action prediction baselines.
• Characterization of post-training reinforcement learning algorithms' applicability regions in shaping planning patterns, revealing on-policy distillation (OPD) has broader and more consistent effective regions than group relative policy optimization (GRPO).
• Multi-Teacher On-Policy Distillation (MOPD) method for integrating multiple teacher policies by converging to shared planning pattern distributions, supporting cross-environment generalization and continual learning with pattern compatibility analysis.

Datasets

• Synthetic Multi-Turn Planning Dataset — 1.2B tokens pre-training corpus plus post-training and test sets — public release on Huggingface at MultimodalAgent/TianyiMen_PlanPhys_Datasets

Baselines vs proposed

• Direct action prediction baseline: avg@8 pass rate = 89.5% (short), 46.6% (middle), 22.7% (long) vs world-modeling: 98.9%, 85.9%, 68.5% respectively (Fig 3)
• GRPO post-training effective region smaller than OPD post-training on long-horizon, low-quality data; OPD yields higher performance improvements across these settings (Fig 1,d,e)
• Suboptimal trajectories inclusion drops performance from near 90% to near 0% pass@8 on long-horizon tasks (Section 4.3)
• Multi-teacher OPD enables cross-environment generalization when planning patterns are shared but causes severe catastrophic forgetting when patterns fully conflict (Section 6)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.24720.

Fig 1: An overview of the studying into physics of multi-turn long-horizon planning. It studies the long-horizon

Fig 2 (page 3).

Fig 22: Parameter Dynamics for MOPD.

Fig 21: Planning pattern distributions for 4 Opt.:4 Sub. (d) before and after multi-teacher OPD.

Limitations

• Experiments conducted on synthetic benchmark environments which, while controllable, may not fully transfer to real-world unstructured data or tasks.
• No explicit adversarial evaluation or robustness testing against malicious perturbations of trajectories or environment.
• Statistical significance of improvements and confidence intervals not reported, limiting assessment of result variability.
• Limited analysis on hyperparameter sensitivity or model size scaling beyond fixed Qwen2.5 architecture.
• Post-training focus on OPD and GRPO omits exploration of other RL or distillation algorithms.
• Multi-teacher distillation assumed ideal teacher policies, but real-world diverse teachers might present greater conflict not covered.

Open questions / follow-ons

• How do the conclusions drawn from controlled synthetic environments generalize to real-world, large-scale, noisy Internet data and open-domain planning tasks?
• Can on-policy distillation be combined with other advanced RL or learning paradigms to improve distillation of new and conflicting procedural knowledge without impairing prior world modeling?
• What approaches can mitigate catastrophic forgetting during multi-teacher integration when planning patterns strongly conflict?
• How sensitive are the findings to variations in model architectures, tokenizer designs, or environment complexity levels?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this paper offers deep insights into how long-horizon sequential decision-making capabilities develop and can be enhanced through controlled training regimes. While CAPTCHA tasks often entail adaptive challenges over multiple interactions, understanding how multi-turn planning abilities are acquired, shaped, and consolidated in foundation models helps in designing better challenge-response mechanisms that adapt dynamically and resist exploitation over time. The study’s controlled environment framework could inspire synthetic CAPTCHA testbeds to systematically test attacker planning abilities and refine challenge difficulty. Additionally, the distinctions between planning patterns and task-specific knowledge underscore the complexity in transferring or distilling automated solver skills across related but distinct CAPTCHA variants, informing defensive upgrades.

Cite

@article{arxiv2607_24720,
  title={ The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation },
  author={ Tianyi Men and Zhuoran Jin and Kang Liu and Jun Zhao },
  journal={arXiv preprint arXiv:2607.24720},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.24720}
}

Read the full paper

• arXiv:2607.24720 (abstract)
• arXiv:2607.24720 (PDF)