Probe-and-Refine Tuning of Repository Guidance for Coding Agents

Source: arXiv:2606.20512 · Published 2026-06-18 · By Asa Shepard, Jeannie Albrecht

TL;DR

This paper addresses the challenge that large language model (LLM)-based coding agents face when working in code repositories without explicit higher-level operational guidance. Such guidance includes knowledge of subsystem file boundaries, test-run instructions, and historical failure patterns, which are typically documented by engineers in AGENTS.md files but whose utility has been contested in recent work. The authors introduce probe-and-refine tuning, a lightweight iterative procedure that uses synthetic bug-fix probes to diagnose and incrementally improve repository guidance by single-shot LLM calls without any multi-step agent loops or reinforcement learning during tuning. Evaluated on SWE-bench Verified over four independent trials with Qwen3.5-35B-A3B, probe-and-refine achieves a 33.0% mean resolve rate, significantly outperforming a static knowledge base initialization (28.3%) and unguided baseline (25.5%), with p < 0.001. The gains primarily arise from increased coverage of evaluable patches (14.5 percentage points more) rather than better patch precision, indicating improved guidance helps agents reach the correct code locations rather than changing patch correctness rates. Further analysis explores budgeted agent steps showing guidance enables productive use of larger step budgets and demonstrates that the tuning loop depends on model capacity since a smaller model degrades the tuning effect though patch precision remains stable. Overall the paper highlights that carefully produced, iteratively refined repository guidance catalyzes broad behavioral improvements in coding agents beyond generic or initial static guidance.

Key findings

• Probe-and-refine guidance improves mean patch resolve rate on SWE-bench Verified to 33.0% ±1.8%, versus 28.3% ±1.4% static knowledge base and 25.5% ±2.2% no guidance baseline (p < 0.001).
• Improvement comes mainly from increased coverage: probe-refine produces evaluable patches for 14.5 percentage points more instances while per-patch precision remains roughly constant (~59%, p = 0.119).
• The refined guidance artifacts average 2,754 characters, 63% longer than static knowledge base artifacts (1,687 chars), with additions largely repo-specific procedural rules and structural knowledge.
• The probe-and-refine tuning uses 3-5 iterations with 10 synthetic probes per iteration, all via single-shot LLM calls, refining static guidance into specialized operational instructions.
• Under larger step budgets, improved guidance enables the coding agent to utilize extra steps productively; without guidance, increasing step budgets yields saturating returns.
• Cross-model experiments show that capacity-constrained models (e.g., NVIDIA Nemotron-3-Nano-30B-A3B) produce insufficient diagnostic output, degrading the tuning loop, although patch precision remains stable.
• Fallback patch generation rates within 200 steps drop substantially under probe-refined guidance (14.8%) versus static-KB (30.8%) and no context (25.6%), indicating better in-loop success.
• Added guidance lines fall into three categories: procedural guardrails (47%), repo-specific structural navigation (30%), and output quality gates (23%), highlighting agent primary failure modes as diagnosis and navigation.

Threat model

Not a security paper; the paper does not specify an explicit adversary model. The implicit challenge is improving coding agent behavior given imperfect or incomplete repository guidance. The adversary is effectively the challenge posed by missing operational knowledge that causes agents to apply fixes in the wrong files or produce ineffective patches. No attacker capabilities or knowledge beyond bug-fix failures are assumed.

Methodology — deep read

1. Threat Model & Assumptions: The adversary here is implicit—potentially buggy or incomplete operational guidance—that may mislead coding agents. The focus is on improving instructions to the agent, not adversarial attacks. The agent uses a fixed coding scaffold and LLM (Qwen3.5-35B-A3B), with a limited 16k token context window and step budget of 200. No direct adversary model; the study tests guidance quality impact.

2. Data: The evaluation uses SWE-bench Verified, with 500 instances drawn from 12 popular Python repositories pinned at specific commits. The dataset contains real GitHub bug-fix issues with held-out test suites. Guidance artifacts were generated once per repository and reused. Synthetic probes are generated dynamically for tuning, distinct from evaluation instances, preventing contamination.

3. Architecture/Algorithm: Probe-and-refine tuning iteratively refines a static natural-language knowledge base (constructed from tree-sitter parsing plus generic best-practice instructions) into repo-specific operational guidance. Each iteration consists of 4 single-shot LLM calls: (a) Generate a batch of 10 synthetic bug-fix probes from the repo at temperature 0.9, targeting diverse subsystems and failure modes; (b) Attempt fixes for each probe using current guidance in a single-shot call; (c) Judge each attempt in single-shot calls to detect success or failure and propose edits; (d) Aggregate per-probe diagnostics and edits into a final set of up to 5 modifications applied mechanically to the guidance text, enforcing a 3000-character limit. This process runs for 3-5 iterations or stops early if guidance stabilizes. The LLM used for tuning is the same as the coding agent's, ensuring consistency. The final product is a concise (≤3000 chars) natural-language guide with detailed procedural, structural, and quality gate rules.

4. Training Regime: There is no parameter updating or gradient tuning; the process solely uses single-shot LLM queries. Each iteration requires ~22 calls (10 probe generation + (10*2) solve & judge + 1 aggregate). The procedure took place under reproducible inference settings with temperature 0.0 for all calls except probe generation to balance variant probe sampling and stable tuning calls. Hardware details are not explicit but presumably standard inference GPUs sufficient for 35B parameter Qwen 3.5.

5. Evaluation: Performance is measured by resolve rate on SWE-bench Verified—number of instances where the agent's patch passes the held-out test suite—evaluated across 4 independent trials with 500 instances each under 3 guidance conditions: no_context, static knowledge base, and probe-refine. A mixed-effects logistic regression accounts for trial and instance effects when testing significance. Secondary metrics include coverage (percentage of attempts producing evaluable patches) and patch precision (percentage of evaluable patches correctly fixing the issue). Ablations include a step-budget sweep showing interaction effects and cross-model generalization tests. Statistical significance is reported with p-values and confidence intervals.

6. Reproducibility: The procedure is based on open-weight Qwen3.5-35B-A3B (weights available). Code for probe-and-refine tuning is publicly released (https://github.com/asashepard/probe-and-refine-tuning). The SWE-bench Verified dataset is public, though pinned commits per repo are specified for consistency. Guidance artifacts are regenerated per repository and fixed across experiments. No training or model weight updates are performed, easing reproducibility.

Example end-to-end: For each iteration, 10 synthetic bug-fix tasks are generated by the LLM for the particular codebase. Each task is passed through a single-shot prompt including current repository guidance to generate a patch candidate, followed by a single-shot judge prompt that outputs success/failure and per-probe patch diagnostics. These diagnostics are then aggregated into a guidance correction prompt that proposes edits to improve the guidance text, which are applied mechanically and trimmed to fit a 3000-character limit. After 3-5 such rounds, the resulting guidance is injected into the coding agent scaffold evaluated on SWE-bench Verified issues from that repository, measuring patch correctness under held-out tests at a 200-step allowance.

Technical innovations

• Introduction of probe-and-refine tuning: a lightweight, iterative procedure that uses synthetic bug-fix probes and single-shot LLM calls to refine repository guidance into operational, repo-specific instructions without parameter tuning or multi-step agent loops.
• Use of a diagnostic feedback loop from synthetic bug-fix probe attempts and judging calls to detect guidance shortcomings and propose targeted incremental edits.
• Empirical demonstration that improved guidance improves coverage (agent reaching the correct files and producing evaluable patches) rather than patch precision, highlighting the importance of operational navigation over patch correctness improvements.
• Detailed analysis of guidance content evolution showing added procedural workflows, repo-structural navigation, and quality gates that capture developer-like operational knowledge absent from generic guidance files.
• Cross-model generalization analysis revealing that the tuning loop depends on model capacity and diagnostic generation quality; guidance tuned for one model can destabilize others without sufficient diagnostic output.

Datasets

• SWE-bench Verified — 500 instances per trial — publicly available benchmark of real GitHub Python repo bug fixes
• Synthetic bug-fix probes — dynamically generated per repo during tuning — internal and distinct from SWE-bench evaluation

Baselines vs proposed

• No Context baseline: mean resolve rate = 25.5% ± 2.2%
• Static Knowledge Base (static_kb): mean resolve rate = 28.3% ± 1.4%
• Probe-and-Refine tuned guidance: mean resolve rate = 33.0% ± 1.8% (p < 0.001 vs both baselines)
• Coverage: probe-refine coverage 56.2% ± 1.8% vs static_kb 50.6% ± 2.3% and no_context 41.7% ± 2.9%
• Precision per patch stable across conditions (~59%, p=0.119), indicating gains come from coverage, not correctness improvements.
• Fallback patch rates within 200 steps decrease under probe-refine guidance (14.8%) compared to static_kb (30.8%) and no_context (25.6%)
• Cross-model test with NVIDIA-Nemotron-3-Nano-30B-A3B shows degradation of tuning loop effectiveness though precision remains stable.

Limitations

• The procedure relies on synthetic probes generated by the same model that consumes the guidance, which may limit generalization across models or evolve guidance biased to a particular model's diagnostic style.
• Guidance artifacts have a fixed character limit (≤3000) which may not scale effectively to very large repositories or complex workflows as noted by related work.
• No explicit adversarial or distribution shift robustness testing; guidance effectiveness under real-world noisy or malicious repository guidance is unknown.
• The tuning loop assumes the model can produce sufficiently diagnostic assessment outputs; degraded performance was observed with smaller-capacity models.
• The reliance on single-shot LLM calls simplifies tuning but may miss deeper multi-turn emergent behaviors that more complex agent loops or reinforcement learning might capture.
• Absolute resolve rates remain modest (<35%) due to context window truncations and step budget limitations, limiting conclusions about ultimate achievable correctness gains.

Open questions / follow-ons

• Can probe-and-refine tuning be adapted to produce scaffolds or other structured representations beyond text guidance, potentially improving interpretability or modularity?
• How does the probe-and-refine procedure perform when scaled to much larger repositories or multi-language codebases, where operational knowledge is more fragmented?
• Can multi-turn or multi-agent collaborative refinement loops improve guidance quality over single-shot calls, and if so, how to balance complexity and benefits?
• What mechanisms can enable cross-model guidance portability or domain adaptation when diagnostic output styles differ significantly between models?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners deploying coding agents or automated fix tools, this paper highlights the critical importance of repository-level operational instructions beyond mere code context. The probe-and-refine approach offers a principled method to improve guidance files iteratively using synthetic tasks, boosting agent coverage without sacrificing precision. This suggests that investment in iterative, failure-informed refinement of agent instructions can yield real performance gains even without upgrading underlying LLM capability or scaffold design. Moreover, understanding how guidance length, quality, and model capacity shape effectiveness can inform design choices when integrating language models into secure software workflows. Finally, the insights about balancing evaluation step budgets and guidance specificity may be applicable to other agentic contexts, including security automation pipelines where mistake-prone agents interact with large technical contexts.

Cite

@article{arxiv2606_20512,
  title={ Probe-and-Refine Tuning of Repository Guidance for Coding Agents },
  author={ Asa Shepard and Jeannie Albrecht },
  journal={arXiv preprint arXiv:2606.20512},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.20512}
}

Read the full paper

• arXiv:2606.20512 (abstract)
• arXiv:2606.20512 (PDF)