OpenForgeRL: Train Harness-native Agents in Any Environment

Source: arXiv:2607.21557 · Published 2026-07-23 · By Xiao Yu, Baolin Peng, Ruize Xu, Hao Zou, Qianhui Wu, Hao Cheng et al.

TL;DR

OpenForgeRL addresses a critical gap in training advanced AI agents that operate through complex inference harnesses—such as Claude Code, Codex, and OpenClaw—which manage multi-turn reasoning, tool use, and external system access. These harnesses create stateful, multi-process environments that typical open reinforcement learning (RL) infrastructures cannot natively support, forcing researchers to use simplified training approximations that cause train–deploy mismatches. OpenForgeRL introduces a scalable, open-source framework that decouples training from inference by employing a lightweight proxy to intercept model calls from any harness and record them as RL training data, plus a Kubernetes-based orchestrator that runs rollouts remotely in containerized environments on cloud infrastructure. This enables true end-to-end training of harness-native agents in diverse, complex environments ranging from text-based tool or claw tasks to multimodal GUI environments (browser and computer use).

Key findings

• OpenForgeClaw trained with hundreds to a few thousand tasks achieves 31.7 pass@3 and 55.9 pass@3 on ClawEval, 33.7 pass@1 on QwenClawBench, and 28.1 pass@1 on MCPAtlas benchmarks, outperforming open baselines of similar size.
• OpenForgeGUI with ~2.5k tasks reaches 37.7 pass@1 on OSWorld-Verified, 63.0 pass@1 on Online-Mind2Web, and 72.3 pass@1 on WebVoyager, matching or surpassing models several times larger, despite using 10x fewer tasks than some baselines.
• SFT+RL training substantially improves performance and agentic reliability over SFT alone, for example increasing pass@3 on ClawEval from 21.7 to 31.7 with OpenForge-Claw.
• Harness choice critically impacts learning difficulty and performance: simpler harnesses with better tool alignment (e.g., ZeroClaw) achieve higher performance than more complex ones (e.g., OpenClaw, Codex).
• Models trained on multiple harnesses generalize better to unseen harnesses, with combined training on ZeroClaw, OpenClaw, and Codex improving pass@1 by up to +20.3 on unseen harnesses.
• RL enhances key agent capabilities including self-verification, tool coverage, and multi-step plan completion, but key challenges such as error recovery remain weak across harnesses.
• The Kubernetes-based rollout orchestrator enables scalable concurrent training on cloud providers (Microsoft Azure) with containerized isolated sandboxes for each environment-harness pair.
• The proxy server abstracts harness inference calls, allowing seamless trajectory reconstruction and integration with standard RL backends like veRL, supporting complex, stateful, multi-turn agent training.

Methodology — deep read

The paper focuses on enabling end-to-end training of harness-based agents operating in complex environments through a Markov Decision Process (MDP) formalism ⟨S, A, T , R, γ⟩ where the agent receives observations (states), takes actions, and receives terminal rewards for task success. The core challenge is that harnesses encapsulate complex multi-turn inference, tool use, and internal state management, breaking assumptions of open RL frameworks that expect simpler single-turn or co-located rollouts.

To overcome this, OpenForgeRL introduces two main components: (1) a lightweight proxy server that intercepts the harness’s model generation calls during rollout, records the prompt-response pairs, and reconstructs standard RL trajectories including rewards; and (2) a Kubernetes-based rollout orchestrator that launches each rollout in a dedicated remote container sandbox on cloud infrastructure (Microsoft Azure), fully decoupled from training nodes and enabling elastic scaling to thousands of parallel environments.

The framework connects any arbitrary agent harness (e.g., Claude Code, Codex, OpenClaw) with any environment (from text/CLI-based tool use like Claw to multimodal GUI browser or computer-use) by wrapping their rollouts within containers and intercepting their model interactions. Training uses off-the-shelf RL codebases such as veRL with group-based policy optimization (GRPO) algorithms.

Data generation involves an automated task synthesis pipeline: it proposes realistic instruction-based tasks grounded in web data and asset pools, prunes low-quality or duplicates, builds verified containerized testing environments with executable files and verification scripts, and tests them end-to-end with open LLM/VLM rollouts. This pipeline produces both supervised fine-tuning (SFT) trajectories from strong teacher models and RL tasks with associated rewards.

Experiments use Qwen3-30B (claw) and Qwen3-VL-8B (GUI) backbone models. SFT pretraining distills successful trajectories from stronger teacher models (MiniMax, Kimi-K2.5). RL fine-tuning uses GRPO with batch and group size of eight, running on 8×B200 GPUs. Rollouts run remotely on Azure container pods via the Kubernetes orchestrator. Metrics include pass@k (success rate) on benchmarks such as ClawEval, QwenClawBench, MCPAtlas for claw; OSWorld-Verified, Online-Mind2Web, and WebVoyager for GUI.

Evaluations include direct benchmark comparison against open baselines and larger closed models, ablations on the effect of SFT vs SFT+RL, analysis of different harnesses’ difficulty, and cross-harness generalization. Statistical tests are not explicitly described. Code, data, and models are planned for release but not yet available. The proxy approach permits seamless integration of real harnesses without needing to simplify or reimplement them for training.

One concrete example: for a claw task (e.g., searching email via a tool call), the harness runs inside a remote container calling the proxy to serve model generations. The proxy records each prompt-response pair (including complex multi-step tool calls managed internally by the harness). Upon completing the rollout and receiving a terminal reward (success/failure), the proxy reconstructs the trajectory and feeds it into the RL backend for policy updates. The Kubernetes orchestrator manages container allocation/scaling transparently.

Overall, the methodology enables training of complex harness-native multi-turn agents in diverse environments at scale without train-deploy mismatch, advancing open research on practical agent deployment settings.

Technical innovations

• A proxy server architecture that intercepts and records harness inference calls, enabling standard RL training pipeline compatibility despite stateful, multi-process harnesses.
• A Kubernetes orchestrator that launches and manages remote containerized rollout environments enabling scalable, decoupled execution of complex harness rollouts disconnected from training nodes.
• An end-to-end automated data synthesis pipeline that generates, verifies, and tests diverse realistic tasks and environments for RL training beyond traditional coding tasks, extending to GUI and browser-use domains.
• Demonstration of cross-harness training and generalization showing that training on multiple complex harnesses simultaneously improves performance and transfer to unseen setups.

Datasets

• SFT Trajectories (Claw): 892
• RL tasks (Claw): 343
• SFT Trajectories (GUI-Computer Use): 795
• RL tasks (GUI-Computer Use): 252
• SFT Trajectories (GUI-Browser): 1496
• RL tasks (GUI-Browser): 900
• Task pools built using web data, ClawHub, ZClawBench, AgentNet, Synthetic-Computer-Use datasets, and filtered WebGym collected tasks.

Baselines vs proposed

• Qwen3-30B-A3B-Thinking: ClawEval pass@3 = 14.3% vs OpenForge-Claw(SFT+RL) = 31.7%
• Qwen3-32B: ClawEval pass@1 = 6.8% vs OpenForge-Claw(SFT+RL) = 33.7% on QwenClawBench
• MolmoWeb-8B: Online-Mind2Web = 35.3% vs OpenForge-GUI(SFT+RL) = 63.0%
• UI-TARS-1.5-7B: OSWorld-Verified = 27.4% vs OpenForge-GUI(SFT+RL) = 37.7%
• OpenCUA-7B: WebVoyager = 66.4% vs OpenForge-GUI(SFT+RL) = 72.3%
• Multi-harness training (ZeroClaw+OpenClaw+Codex) improved unseen harness pass@1 by up to +20.3% over base

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.21557.

Fig 1: Left: OPENFORGE RL builds on Orchard Env (Peng et al., 2026) and connects any harness

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• Error recovery remains a weak ability across all harnesses, indicating that RL training does not fully address robustness to failure.
• Evaluation does not report statistical significance or confidence intervals, making reliability of improvements uncertain.
• The framework depends on proprietary cloud infrastructure (Microsoft Azure) for scalability, potentially limiting accessibility for some researchers.
• Code, data, and models are promised for release but are not yet publicly available, hindering immediate reproducibility.
• The pipeline focuses primarily on known harnesses and environments; generality to novel or radically different harnesses is untested.
• Timeout and error handling strategies are simplistic (discarding partial rollouts) and may lose useful training signal.

Open questions / follow-ons

• How to improve error recovery capabilities of RL-trained agents operating within complex harnesses?
• Can the OpenForgeRL framework support real-time adversarial robustness evaluation or adversarial training in harness environments?
• What are the trade-offs between harness complexity and agent learning efficiency/generalization more broadly?
• Could the proxy-based architecture be adapted for online fine-tuning or continual learning in deployed agent systems?

Why it matters for bot defense

Bot-defense engineers and CAPTCHA practitioners can draw useful lessons from OpenForgeRL's approach to decoupling complex multi-turn inference from training, enabling end-to-end reinforcement learning in rich, stateful agent ecosystems. The proxy and orchestrator design enables practical training of agents that integrate multi-step tool use and environment interaction, which bears analogy to bot architectures that rely on complex layers of external service calls and stateful control flows. Understanding how harness choice impacts learnability and agent reliability may inform better design of multi-component bot detectors or human verification workflows. Furthermore, OpenForgeRL’s scalable cloud-native remote rollout orchestration approach provides a blueprint for deploying and updating sophisticated bot-detection agents operating across diverse input modalities and application contexts. Nonetheless, critical limitations remain, particularly in error recovery and robustness, which are key to real-world bot-defense. Future bot-defense research might explore similar training frameworks to improve detection agents' adaptability, multi-modal environment interaction, and long-horizon behavioral consistency.

Cite

@article{arxiv2607_21557,
  title={ OpenForgeRL: Train Harness-native Agents in Any Environment },
  author={ Xiao Yu and Baolin Peng and Ruize Xu and Hao Zou and Qianhui Wu and Hao Cheng and Wenlin Yao and Nikhil Singh and Zhou Yu and Jianfeng Gao },
  journal={arXiv preprint arXiv:2607.21557},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.21557}
}

Read the full paper

• arXiv:2607.21557 (abstract)
• arXiv:2607.21557 (PDF)