MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesis

Source: arXiv:2607.27146 · Published 2026-07-29 · By Yihao Chen, Shi Chang, Khaled Chawa, Feng Lin, Boyuan Chen, Shaowei Wang et al.

TL;DR

The paper addresses a core challenge in automated software engineering: constructing complete programs from scratch using language models rather than merely modifying existing codebases. Existing benchmarks and training frameworks focus on incremental edits (bug fixes, feature additions), whereas from-scratch synthesis requires a full development life cycle process including specification inference, architecture design, implementation, debugging, testing, and final verification. To tackle this, the authors introduce MindForge, an automated pipeline that transforms open-source command-line programs into source-free training environments exposing only compiled executables and documentation but no source code. Using these environments, they collect whole-life-cycle program synthesis trajectories with a strong teacher agent (GLM-5.2) and use these to fine-tune a smaller model (Qwen3.6-27B). This leads to significant gains in program synthesis performance, raising Qwen3.6-27B’s average pass rate on the 200-instance ProgramBench from 37.98% to 49.51%, a 30.4% relative improvement. These gains also generalize to seven unseen software engineering benchmarks across diverse tasks such as cross-language translation, bug fixing, and feature implementation, with absolute improvements up to 31 percentage points. Their fine-tuned smaller model approaches or surpasses performance of substantially larger frontier models. Behavior analysis reveals that MindForge-trained models exhibit longer and more productive reasoning and editing workflows with lower failure rates despite the increased horizon. Overall, this work demonstrates that scalable source-free training environments combined with whole-life-cycle synthesis trajectories enable smaller language models to substantially improve from-scratch program construction capabilities and generalize broadly within software engineering.

Key findings

• MindForge pipeline converted 562 unique open-source CLI programs into source-free compiled environments across 6 languages, distinct from existing benchmarks.
• Collected 1,001 whole-life-cycle program developmental trajectories using GLM-5.2 teacher agent, with 99.1% coverage of specification exploration and 87.1% coverage of design phases.
• Fine-tuning Qwen3.6-27B on MindForge trajectories improved ProgramBench average test pass rate from 37.98% to 49.51%, a 30.4% relative increase, surpassing larger DeepSeek V4 Pro (47.80%) and nearing GLM-5.1 (50.9%).
• MindForge-27B outperforms base model on 152 of 200 ProgramBench instances (76%), underperforms on 43 (21.5%), ties on 5 (2.5%).
• On seven unseen benchmarks, absolute gains over base Qwen3.6-27B range from +4.94 points (FeatBench) to +31.00 points (RepoZero C2Rust), showing broad generalization.
• Behavioral metrics show MindForge-27B doubled average trajectory length (344 to 736 turns), doubled tool calls (174 to 373), and increased token consumption 5.7× (10.13M to 58.22M per instance) compared to base, indicating more sustained reasoning.
• Command failure rate decreased from 10.98% to 9.35% despite longer synthesis horizon, showing increased operational reliability.
• Rates of converting reasoning or failure recovery actions into implementation edits approximately doubled (from ~28%/32% to ~50%/49%), narrowing gap to frontier models like GLM-5.2.

Methodology — deep read

1. Threat Model & Assumptions: The adversary here is not an attacker but a model attempting to synthesize programs from scratch without source code access. The environment hides internal source and exposes only compiled binaries and documentation, simulating a source-free setting. The agent cannot query the internet or external services during evaluation or training runs.

2. Data: The authors curated 2,235 candidate CLI repositories from public "awesome CLI" GitHub collections, pinned to fixed commits. After automated offline screening, build discovery (automated build script generation), reproducibility checks, and source-free binary verification, 562 unique programs across 6 compiled languages (Go, Rust, C, C++, Swift, TypeScript) remained. Using GLM-5.2 teacher agent, 1,001 whole-life-cycle trajectories were collected, covering specification inference, design, implementation, bug localization and fixing, verification, and refinement phases. Trajectories averaged 181.6 turns and 177K tokens.

3. Architecture / Algorithm: The base student model is Qwen3.6-27B, a 27B parameter LLM. Training updates all language model weights; vision modules remain frozen. Instructor trajectories from GLM-5.2 (754B parameters) serve as teacher demonstrations. The pipeline involves training the student to reproduce entire program synthesis sessions end-to-end, not isolated actions.

4. Training Regime: Fine-tuning used MS-Swift with Megatron backend, batch size 96, microbatch 1, trained for 8 epochs. AdamW optimizer with learning rate 4e-5 warmed up over 10% steps and cosine decay to 4e-6, weight decay 0.04, gradient clipping. Loss computed only over assistant reasoning, natural language and tool calls; system/user messages and outputs masked. Seed fixed at 1105.

5. Evaluation Protocol: Evaluation on ProgramBench 200-instances and seven unseen benchmarks with native metrics (fraction of test cases passed). The student and base models were compared in identical conditions with 512K context windows; Internet access disabled during eval to prevent leakage. Statistical significance tested (p < 0.05). Repository overlaps analyzed to avoid data contamination; minimal overlap found. For long-horizon benchmarks, single run evaluations; multiple runs for smaller.

6. Reproducibility: The paper reports release of the full pipeline including environment construction, trajectory refinement/recovery, and distilled model weights (MindForge-27B). Detailed appendices cover environment creation, recovery of interrupted trajectories, reasoning rewrite to fix narrative coherence without altering underlying actions, and command classification. One concrete example of failure recovery is detailed in Appendix E.1.

End-to-end example: A sample trajectory starts with the agent inferring the specification from the executable and docs, proceeds to design a program architecture, generates initial code, iteratively builds and tests, localizes bugs found at runtime, repairs implementation and tests again, refining until the compile.sh script produces a passing build. Each tool call and reasoning step is recorded and refined for consistency. This full trajectory forms the training signal for the student model.

Technical innovations

• MindForge pipeline automates construction of source-free executable environments from open-source CLI programs, exposing only compiled binaries and documentation, enabling scalable from-scratch program synthesis training.
• High-quality whole-life-cycle program synthesis trajectories are collected at scale with GLM-5.2 teacher agent inside these environments, covering diverse development stages including specification inference and debugging.
• Trajectory refinement techniques including infrastructure-noise recovery and reasoning rewrite ensure training data coherence by repairing premature failures and narrative discontinuities without altering environment interactions.
• Distillation of large teacher trajectories into a smaller 27B parameter student model (Qwen3.6-27B) yields substantial performance improvements, surpassing larger frontier models on ProgramBench and generalizing to multiple out-of-distribution software engineering benchmarks.

Datasets

• MindForge environments — 562 unique programs across 6 compiled languages — constructed from curated 'awesome CLI' GitHub repositories disjoint from ProgramBench
• ProgramBench — 200 instances CLI programs in compiled languages with execute-only binaries and documentation — public benchmark
• Seven unseen software engineering benchmarks (e.g., RepoZero C2Rust 200 tasks, DeepSWE 113 tasks, NL2Repo 104 tasks, SWE-bench variants, FeatBench) — public benchmarks, zero overlap with training data

Baselines vs proposed

• Qwen3.6-27B base: average ProgramBench pass rate = 37.98% vs MindForge-27B (fine-tuned): 49.51%
• DeepSeek V4 Pro: 47.80% vs MindForge-27B: 49.51%
• GLM-5.1: 50.90% vs MindForge-27B: 49.51%
• Opus 4.7: 51.38% vs MindForge-27B: 49.51%
• On RepoZero-C2Rust: base 47.00% vs MindForge-27B 78.00%
• On DeepSWE: base 1.76% vs MindForge-27B 15.92%
• On NL2Repo-Bench with tests: base 61.27% vs MindForge-27B 71.97%
• On SWE-bench Verified: base 68.80% vs MindForge-27B 73.84%
• On FeatBench: base 50.10% vs MindForge-27B 55.05%

Limitations

• The teacher agent GLM-5.2 remains very large (754B parameters), potentially limiting scalability of trajectory collection.
• Program synthesis trajectories rely on successful build completions; failure to pass all tests means some supervision signals may be imperfect or incomplete.
• Evaluation on some large-scale benchmarks was single-run due to computational cost, limiting statistical robustness for those cases.
• Despite improved behavioral alignment, MindForge-27B still lags behind strongest frontier models on absolute pass rates and implementation editing efficiency.
• The pipeline focuses on command-line CLI programs with compiled executables, limiting applicability to other software types or rich interactive systems.
• Internet and external resource access was disabled in evaluation but was not explicitly modeled in training, so real-world robustness to external dependencies remains untested.

Open questions / follow-ons

• How does the MindForge training pipeline scale when using teacher agents smaller than GLM-5.2, or with less computational resources?
• Can the approach be extended from CLI programs and compiled languages to interactive, interpreted, or web-based applications where source-free environments are harder to generate?
• What improvements in student model architecture or training regime could further reduce the gap to the strongest frontier models?
• How robust is the learned model to genuinely novel software tasks that differ significantly from the training environments?

Why it matters for bot defense

This work is highly relevant to bot-defense and CAPTCHA practitioners interested in automated agent behaviors involving multi-step reasoning and complex environment interaction. The MindForge pipeline demonstrates a scalable methodology to train smaller language models to perform whole-life-cycle program synthesis in source-free executable environments. Analogous concepts could inspire design of CAPTCHAs or bot-resilience mechanisms that require multi-turn, multi-modal problem-solving against obfuscated or black-boxed challenge environments. The trajectory refinement techniques also highlight how to build training datasets capturing rich agent reasoning and error recovery, which could improve behavioral detection and anomaly classification in security contexts. Additionally, the analysis of failure-recovery transitions and behavioral editing rates informs how to interpret agent persistence versus noise, useful for bot defense signal engineering. However, direct use of such models to automatically bypass security challenges also suggests a red-team risk. Practitioners should consider the scaling of model capabilities from this research when assessing future risks to CAPTCHA robustness posed by advanced small models trained under realistic black-box conditions.

Cite

@article{arxiv2607_27146,
  title={ MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesis },
  author={ Yihao Chen and Shi Chang and Khaled Chawa and Feng Lin and Boyuan Chen and Shaowei Wang and Ahmed E. Hassan },
  journal={arXiv preprint arXiv:2607.27146},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.27146}
}

Read the full paper

• arXiv:2607.27146 (abstract)
• arXiv:2607.27146 (PDF)