DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment

Source: arXiv:2607.07820 · Published 2026-07-08 · By Xinyu Geng, Xuanhua He, Sixiang Chen, Yanjing Xiao, Fan Zhang, Shijue Huang et al.

TL;DR

This paper addresses the challenge of training autonomous web agents that can improve their tool-use abilities from their own experience without relying on stronger external teacher models. Prior approaches like supervised fine-tuning on fixed distilled trajectories saturate quickly, while reinforcement learning suffers from sparse and coarse reward signals that do not expose failures at intermediate steps. To overcome these limitations, the authors introduce DeepSearch-World, a deterministic and verifiable offline environment built on a 10M-article Wikipedia corpus, which supports reproducible, multi-hop QA tasks with entity-level random-walk-based queries. This environment supports key cognitive behaviors such as progress verification, grounded reflection on failure, and failure recovery by enabling interaction step verification. On top of this, they propose DeepSearch-Evolve, a self-distillation training framework that iteratively generates verified trajectories using a scaffold teacher agent, filters and converts these into ReAct-format training data, and fine-tunes the agent in a stable evolving supervised fine-tuning loop. The final DeepSearch-World-9B agent trained solely on its own verified rollouts achieves competitive or superior performance compared with strong open-source deep search agents—reaching 31.2% on BrowseComp, 61.5% on GAIA, and 93.4% on HotpotQA benchmarks. This demonstrates scalable self-improvement of long-horizon web agents without requiring stronger teacher models or externally synthesized data.

Key findings

• DeepSearch-World contains 420K multi-hop QA tasks constructed from entity-level random walks and an aligned offline Wikipedia corpus of 10M articles.
• The environment supports deterministic and verifiable tool calls for search (BM25 retrieval) and document visit (SQLite offset lookups), enabling entity-level progress verification after each step.
• DeepSearch-Evolve iteratively generates verified tool-use trajectories, applies rejection sampling and a Qwen3.5-9B-based trajectory quality filter, converts scaffolded supervision to standard ReAct format, and fine-tunes the agent with evolving supervised fine-tuning.
• DeepSearch-World-9B improves over baseline Qwen3.5-9B-Instruct by +23.8% on BrowseComp, +37.6% on GAIA, +48.1% on HotpotQA, demonstrating strong gains in complex reasoning tasks.
• Compared to Qwen3.5-9B-Instruct, DeepSearch-World-9B executes longer average tool-use episodes (18.0 vs 4.7 interaction rounds), more search calls (12.6 vs 3.8), and more visit calls (5.4 vs 0.9), reflecting better multi-step evidence gathering.
• Ablation studies show that combining answer-correctness rejection sampling and trajectory quality filtering improves SearchQA accuracy from 46.4% to 58.2%.
• Rewriting environment reflections during scaffold-to-ReAct conversion is critical, improving DeepSearch-Val accuracy from 16.7% to 31.9%.
• The evolving SFT framework benefits from a large diverse data pool (420K vs 100K QA instances), leading to higher validation scores, reduced format errors, and improved tool success rates.

Threat model

N/A — this paper does not directly address adversarial threats but focuses on environment design and stable self-distillation for tool-use agents. The environment assumes a deterministic verifier that prevents the agent from cheating or corrupting observations, enabling grounded reflection and failure detection.

Methodology — deep read

The methodology centers on constructing a deterministic and verifiable offline environment called DeepSearch-World, and using a self-distillation training framework DeepSearch-Evolve for tool-use web agents.

1. Threat Model & Assumptions: The adversary is not explicitly modeled as this is a training environment and methodology paper rather than a security one. The environment assumes deterministic tool calls where observations and progress can be accurately verified. The agent cannot cheat or manipulate the environment; failures and progress are objectively tracked.

2. Data: The environment builds upon the full Wikipedia hyperlink graph (around 10 million articles). A large dataset of 420K multi-hop QA tasks is constructed by sampling multi-hop random walks over entities in the graph and converting these into obfuscated multi-hop questions requiring multi-step reasoning over linked Wikipedia pages. A held-out set of 377 high-quality QA tasks (DeepSearch-Val) is reserved for validation. Answers are human-verified by a 5-expert pool.

3. Architecture / Algorithm: Tool-use agents operate in the environment via two deterministic tools—search (a BM25-based retrieval function returning page snippets and URLs) and visit (retrieving full article text by URL). The scaffold teacher agent formalizes tool-use as a three-phase process: PLAN, ACT, and END. The plan state tracks completed, todo subgoals, failures, and collected information. ACT iteratively calls tools, updates state, and uses environment feedback to reflect and recover from failures. END aggregates the final grounded answer.

The self-distillation framework DeepSearch-Evolve iteratively generates verified trajectories using the scaffold teacher, applies answer correctness and trajectory quality filters (using Qwen3.5-9B as a judge), converts scaffolded trajectories into standard ReAct supervision by rewriting progress states and reflections, then fine-tunes a student agent in a supervised manner using standard negative log likelihood loss on these trajectories. This produces a stable, partially on-policy evolving supervised fine-tuning loop rather than full on-policy reinforcement or token-level distillation.

4. Training Regime: The backbone is Qwen3.5-9B. 11 rounds of self-evolving training were run, each generating trajectories for 10K new QA instances and fine-tuning once 4K passed filtering. Trajectory generation was capped at 30 steps. An asynchronous generation and training scheme was used with importance sampling past data to mitigate forgetting. Each fine-tuning round runs for one epoch using Llama Factory. To bridge offline and real tool execution gaps, 1,600 real-tool instances with Google SerpAPI and Jina retrieval were fine-tuned on with a reinforcement learning policy optimization method (GRPO).

5. Evaluation Protocol: Models were evaluated on 7 deep search and reasoning benchmarks including BrowseComp, GAIA, HotpotQA, among others. Main metrics were accuracy / exact match percentages. Baselines include open-source and proprietary recent agents. Ablations studied the impact of rejection sampling, trajectory filtering, scaffold-to-ReAct conversion, and data pool size. Tool-use metrics like average episode length, number of search and visit calls, and an advanced capability score judged by an LLM were also analyzed.

6. Reproducibility: The authors commit to releasing the offline environment, the 420K QA dataset, validation splits, the model weights for DeepSearch-World-9B, and code. The dataset and environment are deterministic and grounded in static Wikipedia snapshots to allow fully reproducible experimentation.

As an example, a single training round proceeds as follows: The current agent acts as teacher and generates trajectories for 10K unseen QA tasks using planned search and visit tool calls, leveraging environment feedback to reflect and retry queries. These trajectories are filtered for answer correctness and quality. The remaining trajectories are converted from scaffolded format to ReAct-style training data with progress states and reflections rewritten as reasoning tokens. The agent is then fine-tuned for one epoch on this data, producing a stronger agent for the next round. Iterating this process for 11 rounds leads to steadily improving performance and more sophisticated multi-step tool use.

Technical innovations

• Construction of DeepSearch-World: a large-scale deterministic offline Wikipedia environment with verifiable entity-level multi-hop QA, enabling reproducible tool-use supervision.
• DeepSearch-Evolve self-distillation framework that iteratively generates, filters, converts, and fine-tunes agent trajectories in a scaffold-to-ReAct pipeline for stable evolving supervised fine-tuning.
• Grounded reflection mechanism during teacher rollout that uses environment verification to detect failure and produce staged query reformulation and failure recovery signals.
• A trajectory quality filtering approach combining answer correctness verification with Qwen3.5-9B prompt-based consistency checks to ensure high-quality training trajectories without stronger external teachers.
• Use of scaffold teacher agent with explicit planning, memory, and error recovery states converted into standard ReAct training data, capturing multi-step tool-use cognition in the learned policy.

Datasets

• DeepSearch-World QA pool — 420K multi-hop QA tasks over Wikipedia entities — constructed from Wikipedia hyperlink graph random walks
• DeepSearch-Val — 377 held-out multi-hop QA instances for validation — human-verified answers in DeepSearch-World environment

Baselines vs proposed

• Qwen3.5-9B-Instruct backbone: BrowseComp accuracy = 7.4% vs DeepSearch-World-9B = 31.2%
• Qwen3.5-9B-Instruct: GAIA = 23.9% vs DeepSearch-World-9B = 61.5%
• Qwen3.5-9B-Instruct: HotpotQA = 45.3% vs DeepSearch-World-9B = 93.4%
• OpenAI Deep Research (proprietary): BrowseComp = 51.5% vs DeepSearch-World-9B = 31.2% (competitive without proprietary teacher)
• Rejection sampling only (RS): SearchQA = 54.9% vs no filtering = 46.4%
• Trajectory quality filtering only (QF): SearchQA = 48.1% vs no filtering = 46.4%
• RS + QF: SearchQA = 58.2%, outperforming OPSD (49.0%) and Skill-SD (47.8%) self-distillation baselines
• Full self-distillation pipeline on DeepSearch-Val: 31.9% accuracy vs vanilla SFT 25.0%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.07820.

Fig 2: Overview of the DeepSearch-World and DeepSearch-Evolve. DeepSearch-World provides a determin-

Fig 3: Three-phase scaffolded teacher rollout.

Fig 1: Conceptual comparison of self-evolving paradigms for tool-use agents. SFT imitates positive traces,

Fig 4: Example scaffold trajectory generated by the teacher agent. The teacher decomposes a fuzzy multi-

Fig 5 (page 3).

Fig 6 (page 3).

Fig 5: Data scale effect in evolving SFT. Compared with 100K total training data, 420K QA lead to a higher

Limitations

• The environment is limited to Wikipedia corpus, restricting knowledge coverage and domain diversity.
• Only evolving supervised fine-tuning is explored; reinforcement learning or on-policy self-distillation (OPSD) could add flexibility but are not fully developed here.
• The current method depends on deterministic, offline environment assumptions not easily extended to real-time or open web settings.
• Cross-lingual generalization is limited as training data is English-only, demonstrated by weaker BrowseComp-ZH results.
• The trajectory filtering and rejection sampling rely on correctness verification from heuristics and a Qwen3.5-9B judge, potentially biased or imperfect.
• Does not address robustness to adversarial tool-use failures or distribution shifts beyond Wikipedia snapshots.

Open questions / follow-ons

• How can the approach be extended to richer domains beyond Wikipedia with more diverse, noisy, or dynamic knowledge sources?
• Can reinforcement learning or OPSD methods be combined with scaffolded supervision to further improve generalization and lifelong adaptation?
• What is the best way to inject higher-level cognitive skills like planning, error recovery, and strategy into RL or end-to-end training frameworks for tool use?
• How can the method handle non-deterministic environments and imperfect tool APIs typical of real web browsing and search?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this work highlights the importance of having a deterministic, verifiable environment to enable stable and scalable agent self-improvement through detailed supervision of intermediate tool-use steps rather than only sparse outcomes. Captchas and bot detection mechanisms could benefit from incorporating components that provide fine-grained stepwise verification signals to detect and thwart automated multi-step tool-use attacks. The trajectory filtering and rejection sampling approach also suggests a model for filtering unlabeled or uncertain agent behaviors in real time. Although developed for web search QA agents, the underlying principle of verifiable multi-step interaction and scaffolded self-distillation could inspire CAPTCHA designs requiring complex chaining of verified interactions, making bots' autonomous improvement harder without human calibration. This approach offers a path toward creating more robust verification signals against evolving information-seeking or automation attacks.

Cite

@article{arxiv2607_07820,
  title={ DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment },
  author={ Xinyu Geng and Xuanhua He and Sixiang Chen and Yanjing Xiao and Fan Zhang and Shijue Huang and Haitao Mi and Zhenwen Liang and Tianqing Fang and Yi R. Fung },
  journal={arXiv preprint arXiv:2607.07820},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.07820}
}

Read the full paper

• arXiv:2607.07820 (abstract)
• arXiv:2607.07820 (PDF)