CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

Source: arXiv:2608.06352 · Published 2026-08-06 · By Fanzhe Meng, Guoxin Chen, Jiale Zhao, Shuang Sun, Zhiyu Lin, Wayne Xin Zhao et al.

TL;DR

CalibForge addresses a key challenge in training large language model terminal agents: constructing terminal tasks that are not only executable and verifiable but also appropriately challenging to foster effective learning. Existing synthesis methods emphasize task validity but fail to ensure tasks lie within a solver-relative difficult yet solvable "learnable zone." CalibForge introduces adversarial solver calibration as a novel autonomous synthesis framework that iteratively revises candidate tasks against solver feedback to position them precisely within this learnable zone. It operationalizes two calibration strategies—multi-solver calibration, which looks for disagreement among heterogeneous solvers, and contrastive solver calibration, which uses a stronger-versus-weaker solver pass/fail relationship to bound task difficulty.

Leveraging these strategies, CalibForge synthesizes 5,431 calibrated terminal tasks that substantially improve training supervision effectiveness. Models fine-tuned on CalibForge tasks outperform state-of-the-art baselines by large margins on Terminal-Bench 2.0 (up to +24.71 points), and also demonstrate strong cross-benchmark transfer gains on long-horizon software engineering tasks (e.g., SWE-bench Pro and Doc2Repo). Ablations confirm that solver calibration distinctly outperforms use of mere validation or single-solver feedback alone, highlighting the value of solver-relative learnability as a construction principle for scaling terminal task training data.

Key findings

• CalibForge constructs 5,431 calibrated terminal tasks spanning 16 diverse domains, avoiding overconcentration seen in prior datasets.
• Models trained on CalibForge tasks achieve 32.58% and 47.57% accuracy on Terminal-Bench 2.0 with Qwen3-30B-A3B-Instruct and Qwen3.5-35B-A3B backbones, respectively, outperforming baselines by up to +6.75 percentage points.
• Cross-distribution transfer improvements include +27.68 percentage points on SWE-bench Pro and +30.04 points on Doc2Repo for the Qwen3-30B-A3B-Instruct backbone.
• Multi-solver calibration yields a +6.74-point improvement in Terminal-Bench 2.0 accuracy over no solver feedback, while contrastive calibration yields a +8.62-point improvement, both significantly better than single-solver feedback (+1.87 points).
• Only 19% of candidates initially satisfy the contrastive calibration pass/fail relation after validation and self-solving, but revision and re-probing increase acceptance to 96%, demonstrating effectiveness of feedback-driven task revision.
• Most candidates are corrected toward the target learnable zone within five calibration rounds, though 7% require 20 or more rounds.
• CalibForge tasks feature richer interaction trajectories with fewer steps but more teacher reasoning tokens than baselines (median 21 steps vs 28 for CLI-Gym; 5.3k vs 4.0k tokens).

Threat model

The paper models an automated adversary in the form of an authoring agent that iteratively revises task specifications informed by multiple solver agents acting as evaluators with varying capabilities. The authoring agent cannot generate tasks that are unsolvable by all solvers or trivially solvable by all; instead, it aims to identify tasks lying in the capability interval bracketed by designated strong and weak solvers. External attackers or malicious solvers are out of scope. Feedback is assumed honest and reliable, and solvers have bounded interaction steps and time limits.

Methodology — deep read

1. Threat Model and Assumptions: CalibForge assumes an adversarial, iterative construction setting where an authoring agent and heterogeneous solver agents interact. The solvers are large language model based subagents with varying capabilities. The goal is to synthesize tasks that lie within a "learnable zone" defined relative to solver strength—tasks must be solvable by at least one solver but not trivially solved by all. The adversary is the automated task author, which aims to produce calibrated tasks, while solver feedback is used as diagnostic feedback to guide task revision. The human adversary capabilities or external attackers are not the focus.

2. Data: Starting from a seed clue (e.g., GitHub issues, documentation), the authoring agent searches web technical sources and develops terminal task specifications including instructions, environment (Docker) setup, and verification tests. Candidate tasks are validated to ensure executability and correctness (structural validation and self-solving). The final curated dataset contains 5,431 tasks after solver calibration and decontamination against benchmark overlaps. Task distribution covers 16 domain categories with checks for diversity in initial artifacts, dependencies, and verifier tests.

3. Architecture / Algorithm: CalibForge comprises an authoring agent (DeepSeek-V4-Pro) and multiple solver subagents (DeepSeek-V4-Flash, GLM-5, Kimi K2.5). The core loop involves repeated probing of candidate tasks by solvers and authoring agent revision guided by detailed feedback including pass/fail outcomes, trajectories, step counts, and self-assessments. Two retention criteria govern calibration: (a) Multi-solver calibration requires at least one solver to pass and at least one to fail, indicating task difficulty lies between solver capabilities. (b) Contrastive solver calibration requires a designated stronger solver to pass and a weaker solver to fail, defining a capability interval. Candidates failing retention criteria are revised and re-validated in up to 50 calibration rounds.

4. Training Regime: Successful tasks produce verified solver trajectories distilled as supervised fine-tuning (SFT) data using DeepSeek-V4-Pro. Training uses Qwen3-30B-A3B-Instruct and Qwen3.5-35B-A3B backbones with full-parameter SFT for 10 epochs. The evaluation uses consistent teacher models, rollout budgets, and task timeout constraints (max 200 steps and 1 hour).

5. Evaluation Protocol: Performance is measured on Terminal-Bench 2.0 primary benchmark and out-of-distribution tests SWE-bench Pro and Doc2Repo. Metrics include task accuracy (pass rates on verification tests). Each benchmark is evaluated with multiple runs and reported with means and standard errors. Benchmark decontamination avoids data leakage by fuzzy matching and heuristic filtering of training candidates.

6. Reproducibility: Authors provide detailed algorithmic description and calibration specifications. Task construction uses publicly available technical Web resources. Baseline task sets are open source. The codebase and raw dataset release are not explicitly stated; follow-up would be needed to confirm open source status. Training and evaluation are performed on large-scale compute with multiple solver backbones.

Example: Starting from a clue—such as a GitHub issue describing a version conflict—the authoring agent constructs a candidate task with instructions, a Docker environment, and verification tests. Structural validation ensures environment builds and tests fail initially. Self-solving attempts by the authoring agent confirm solvability. The candidate then enters multi-solver calibration where DeepSeek-V4-Flash, GLM-5, and Kimi K2.5 solvers attempt the task concurrently. If all pass or all fail, the authoring agent revises instructions, environment, or tests informed by solver trajectories and failure diagnostics. This loop continues until the task satisfies the retention criterion of at least one pass and one fail outcome, securing its retention as a learnable, non-trivial task for training.

Technical innovations

• Formulation of environment-level behavioral calibration that treats terminal task learnability as solver-relative and guides iterative task revision based on verified solver behavior.
• Multi-solver calibration strategy leveraging disagreement across diverse solver agents to identify appropriately challenging tasks beyond executable validation.
• Contrastive solver calibration strategy utilizing a strong-pass/weak-fail relation to define a solver-relative learnable capability interval.
• An autonomous task synthesis loop integrating web research, sandbox environment construction, multi-round solver probing, and feedback-driven revision to scale learnable terminal tasks.

Datasets

• CalibForge terminal task set — 5,431 tasks — synthesized
• SETA-Env — 1,375 tasks — public release (Jan 2026)
• SWE-bench Pro — 731 tasks — public
• Doc2Repo — 731 tasks — public

Baselines vs proposed

• No solver feedback (base) on Qwen3-30B-A3B-Instruct: Terminal-Bench 2.0 accuracy = 22.47% vs CalibForge multi-solver: 29.21%, contrastive solver: 31.09%
• Single-solver feedback baseline on Qwen3-30B-A3B-Instruct: 24.34% vs CalibForge contrastive solver: 31.09%
• TermiGen baseline on Qwen3-30B-A3B-Instruct: 23.60% Terminal-Bench 2.0 vs CalibForge 32.58%
• TerminalTraj baseline on Qwen3-30B-A3B-Instruct: 26.22% Terminal-Bench 2.0 vs CalibForge 32.58%
• Qwen3.5-35B-A3B base: 39.10% Terminal-Bench 2.0 vs CalibForge 47.57%
• TermiGen on Qwen3.5-35B-A3B: 40.07% Terminal-Bench 2.0 vs CalibForge 47.57%
• TerminalTraj on Qwen3.5-35B-A3B: 40.82% Terminal-Bench 2.0 vs CalibForge 47.57%
• SWE-bench Pro improvement on Qwen3-30B-A3B-Instruct: base 3.26% vs CalibForge 30.94%
• Doc2Repo improvement on Qwen3-30B-A3B-Instruct: base 5.94% vs CalibForge 35.98%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.06352.

Fig 1: Overview of CalibForge. Top: Multi-solver and contrastive calibration target a solver-

Fig 2: Overview of CalibForge. Starting from a clue, CalibForge authors a candidate task,

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• The approach relies on the availability of diverse, performant solver agents; results may degrade if solver capabilities converge or lack diversity.
• Calibration uses automated metrics (pass/fail and solver trajectories) but lacks adversarial human evaluation or robustness testing against adversarial task formulations.
• The training is limited to Qwen backbones and particular baseline task sets; generality to other model families requires validation.
• The task synthesis pipeline involves significant compute for up to 50 calibration rounds per candidate, which may limit scalability for very large datasets.
• The paper does not discuss how non-determinism or solver stochasticity affects calibration stability or task retention decisions.
• Code release and dataset accessibility are not fully described, limiting reproducibility at this stage.

Open questions / follow-ons

• How does solver stochasticity or non-determinism in interactions affect the robustness and consistency of the calibration process?
• Can adversarial calibration be extended to incorporate human-in-the-loop feedback or adversarial perturbations to further improve task difficulty tuning?
• How transferable are CalibForge-synthesized tasks and calibration principles to other model families beyond the tested Qwen backbones?
• What is the impact of increasing solver diversity or scaling solver sizes on calibration efficiency and task quality improvements?

Why it matters for bot defense

From a bot-defense or CAPTCHA perspective, the principles underlying CalibForge offer an intriguing paradigm for generating tasks that are neither trivially solvable by automated agents nor completely unsolvable, effectively tuning challenge difficulty relative to attacker capabilities. The notion of adversarial solver calibration could inspire dynamically adjusting CAPTCHA challenges by monitoring multiple solver (bot) behaviors and retaining challenges eliciting solver disagreement or differential pass/fail outcomes. This is potentially more robust than fixed, handcrafted CAPTCHA tasks or simple validation failures.

Further, the construction pipeline emphasizes detailed environmental setup, iterative feedback-driven tuning, and diverse interaction trajectories that could inspire more complex CAPTCHA schemes involving multi-step tasks rather than static puzzles. However, practical adoption would require adaptation to real-time user interaction constraints and mitigation against solver-agent mimicry attacks. Overall, CalibForge contributes a principled framework for calibrating learnability and challenge difficulty relative to solver capabilities, an important conceptual insight for CAPTCHA designers.

Cite

@article{arxiv2608_06352,
  title={ CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks },
  author={ Fanzhe Meng and Guoxin Chen and Jiale Zhao and Shuang Sun and Zhiyu Lin and Wayne Xin Zhao and Ruihua Song and Ji-Rong Wen and Kai Jia },
  journal={arXiv preprint arXiv:2608.06352},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.06352}
}

Read the full paper

• arXiv:2608.06352 (abstract)
• arXiv:2608.06352 (PDF)