Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills

Source: arXiv:2607.22529 · Published 2026-07-24 · By Siyuan Huang, Pengyu Cheng, Haotian Liu, Tao Chen, Yihao Liu, Jingwei Ni et al.

TL;DR

This paper addresses a core challenge in self-evolutionary training of large language models (LLMs): balancing open-ended task diversity with reliable verification of task validity and solution quality. Existing approaches either rely on environment-bound, domain-restricted feedback for precise verification or broad unguided task generation paired with passive post-hoc filtering that suffers from noisy, ill-posed tasks. The authors propose an intermediate abstraction called "skills"—modular, reusable procedural units that encode structural task interfaces and validators. They introduce Skill Self-Play (Skill-SP), a reinforcement learning framework uniting a proposer, solver, and skill controller in a closed-loop co-evolutionary cycle. The proposer generates skill-conditioned tasks, the solver attempts solutions, and the controller refines the skill library based on execution feedback. This maintains rich task diversity and verifiable correctness without context bloat or quality collapse.

Empirical results on diverse tool-use and logical reasoning benchmarks show that Skill-SP consistently improves several LLM backbones, often outperforming unguided self-play and rescuing misaligned models. Gains include up to +42.9 points on tool use (Ministral-3-8B) and +12 points on logic puzzles (Ministral-3-14B), demonstrating the framework’s ability to push both learning frontiers and structural fidelity. Ablations validate the importance of dynamic skill evolution, balanced curriculum construction, and co-evolutionary training of proposer and solver. Overall, Skill-SP enables sustainable, open-ended LLM self-improvement that balances broad exploration and reliable verification through modular skill interfaces.

Key findings

• Skill-SP yields up to +42.9 absolute points improvement on tool-call prediction for initially weak models (Ministral-3-8B) over the base, surpassing unguided self-play gains of +0.9 points on the same.
• On logical reasoning (ZebraLogic), Skill-SP achieves +12.0 point grid-level accuracy gains for Ministral-3-14B, with improvements exceeding +35 points on smaller puzzles.
• Skill-SP consistently outperforms unguided self-play across 5 LLM backbones on API-Bank and BFCL benchmarks, with absolute gains between +2.8 to +6.5 points on competent models.
• Unguided self-play fails to synthesize valid logical reasoning tasks, collapsing training, whereas Skill-SP bootstraps a stable learning loop.
• Ablation studies show static skill sets or uniform routing decrease accuracy by 1.9–2.3 points, demonstrating the necessity of dynamic skill evolution and routing.
• Freezing proposer or feedback solver policies leads to drops of 2.1 and 3.0 points respectively, validating the importance of co-evolutionary updates.
• Skill-only training (without open-ended exploration) degrades generalized evaluation scores, confirming the benefit of mixed skill-conditioned and exploration curricula.
• Skill-SP’s gating of proposer rewards by binary task validity filters prevents reward hacking and ensures structural correctness during task generation.

Threat model

The adversary is internal to the training loop, attempting to maximize proposer reward via task generation. They could produce ill-posed or unsolvable tasks to artificially inflate difficulty and corrupt the curriculum (reward hacking). The system prevents this by gating proposer rewards on strict structural and logical task validity verified by validators and majority consistent solver rollouts. The adversary cannot forge tasks passing these checks unnoticed, nor can they disrupt the solver's co-evolution from frontier feedback.

Methodology — deep read

The paper formalizes a verifiable agent task as a prompt and a hidden verification contract that yields a reward based on solver success. The Skill-SP framework orchestrates three interacting RL policies: a proposer generating tasks conditioned on sampled skills from an evolving skill library; a solver executing solutions; and a skill controller maintaining and evolving the skill library based on execution feedback.

Threat model assumes an internal training loop where tasks must be both diverse to expand the curriculum and structurally valid to avoid reward hacking. The challenge is to jointly evolve task generation and verification without manual annotations.

Data derives from open-ended generations guided by skills (modular procedural interfaces encoding metadata, rules, examples, validators, and usage stats) and an unguided exploration stream without constraints. Skills inject strong structural priors and rigorous validation pipelines enforce three-stage filtering: schema compliance, contract validator success, and multiple rollouts consensus.

The proposer is updated using Group Relative Policy Optimization (GRPO) to maximize a gated reward composed of validity filtering combined with proximity of solver success rate to 0.5 (medium difficulty), encouraging tasks near the solver learning frontier. The solver is trained on a dynamically curated curriculum mixing skill-conditioned and exploratory tasks ranked by this reward.

The skill library continuously evolves: the controller refines existing skills based on failure traces, prunes saturated skills with low frontier reward, and induces novel skills from high-reward exploratory samples using an abstraction policy and lexical novelty checks.

Training relies on repeated iterations of task generation, verification, solver training, proposer policy improvement, and skill library update. Experiments use five LLM backbones (3B–14B parameters), across tool-use benchmarks (API-Bank and BFCL) and logical reasoning (ZebraLogic). Evaluations report accuracy averaged over 8 trials with exact correctness and format compliance as metrics.

A concrete training iteration example: a skill is sampled weighted by historical success and exploration bonuses, guiding the proposer to generate tasks fitted to that skill interface. Generated tasks are verified by validators plus multiple solver rollouts to ensure consistency. Valid high-frontier-reward tasks form a training pool to update the solver via RL. The proposer is optimized to improve future task generation based on updated solver success signals, while the skill controller updates statistics and evolves skills, completing the co-evolution cycle.

Code and experimental details are publicly released for reproducibility, enabling external validation and extension.

Technical innovations

• Introducing modular skill interfaces as reusable, co-evolving procedural units to proactively guide task generation and verification during self-play.
• A closed-loop triadic RL architecture uniting proposer, solver, and skill controller policies for dynamic curriculum construction and skill library evolution.
• A gated curriculum reward combining binary task validity filtering with a medium-difficulty objective to prevent reward hacking and ensure structural task correctness.
• Dynamic skill sampling balancing exploitation of effective skills with exploration bonuses for under-tested skills, maintaining diversity without context bloat.
• Automatic skill library maintenance including refinement from failure diagnosis, pruning of saturated skills, and induction of novel skills from open-ended exploration samples.

Datasets

• API-Bank — multiple difficulty levels (L1-L3) — public/code-benchmark
• BFCL — diverse programming language categories and live scenarios — publicly referenced
• ZebraLogic — logical reasoning puzzles of varying complexity — academic benchmark

Baselines vs proposed

• Base model (Qwen3-4B-Ins) on API-Bank avg: 58.1% tool-call accuracy vs Skill-SP: 64.6% (+6.5)
• Unguided SP on API-Bank avg: 60.2% vs Skill-SP: 64.6% (+4.4)
• Ministral-3-8B base on API-Bank avg: 33.7% vs Unguided SP: 34.4% vs Skill-SP: 61.5% (+27.8)
• Qwen3-4B-Ins base on ZebraLogic logic reasoning grid acc: 72.1% vs Skill-SP: 73.5% (+1.4)
• Ministral-3-14B base on ZebraLogic grid acc: 5.4% vs Skill-SP: 17.4% (+12.0)
• Unguided SP fails to produce meaningful reasoning tasks, thus no evaluation on ZebraLogic
• Uniform routing ablation drops API-Bank overall accuracy by 1.9 points vs full Skill-SP
• Frozen skills ablation drops overall accuracy by 2.3 points, both showing dynamic skill evolution necessity

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.22529.

Fig 1: | Motivation of Skill Self-play. Left: Environment-bound methods provide reliable verification but

Fig 2: | Performance footprint. Skill-

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 3: | Overview of Skill Self-Play. An evolving skill library routes task-generation guidance to the proposer,

Fig 7 (page 4).

Fig 8 (page 4).

Limitations

• Skill-SP requires an initial capability threshold in solver to bootstrap valid training tasks, otherwise training stagnates.
• Progress on largest logical reasoning puzzles (X-Large scale) remains limited for weaker models.
• Experiments focus on a limited set of benchmarks in tool use and logical reasoning; other task families remain unexplored.
• Reliance on predefined validators for task correctness may limit applicability where validators are unavailable or costly.
• Automatic skill induction depends on lexical similarity thresholds for novelty, which may be suboptimal and miss semantically distinct skills.
• The approach may involve computational overhead from co-evolving multiple policies and skill maintenance.

Open questions / follow-ons

• How well does the Skill-SP framework generalize to other complex open-ended domains beyond tool use and logical reasoning?
• Can the automatic skill induction process be improved with semantic embeddings or learned skill representations instead of lexical similarity?
• What are the training and inference efficiency trade-offs of maintaining a large evolving skill library at scale?
• How robust is Skill-SP under adversarial self-play conditions designed to deliberately mislead verification or induce curriculum collapse?

Why it matters for bot defense

For bot-defense and CAPTCHA engineering, Skill-SP highlights how modular skill abstractions can mediate between controlled environment-bound verification and fully open-ended task generation. Applying similar skill-conditioned curricula could help in automatically evolving CAPTCHA challenges with verified security properties, maintaining both diversity and robustness. The framework’s gating of task validity before acceptance into training loops addresses a common bot-defense dilemma of balancing challenge novelty with attack surface reliability. Incorporating co-evolution of challenge generation and solver models guided by modular skills might enable sustainable improvements in adaptive CAPTCHA difficulty calibration, mitigating repetitive pattern exploitation by bots. However, the computational complexity and requirement for rigorous validators pose practical challenges for real-time CAPTCHA deployment.

Cite

@article{arxiv2607_22529,
  title={ Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills },
  author={ Siyuan Huang and Pengyu Cheng and Haotian Liu and Tao Chen and Yihao Liu and Jingwei Ni and Shijie Zhou and Ziyi Yang and Gangwei Jiang and Mengyu Zhou and Yu Cheng and Xiaoxi Jiang and Guanjun Jiang },
  journal={arXiv preprint arXiv:2607.22529},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.22529}
}

Read the full paper

• arXiv:2607.22529 (abstract)
• arXiv:2607.22529 (PDF)