Agentic Synthesis against Counterexample-Supplemented Sketches

Source: arXiv:2607.15854 · Published 2026-07-17 · By Muness Castle, Eric Rubeck

TL;DR

The paper addresses a key challenge in coding agent workflows where automatic repairs fixing failing test cases may not preserve the underlying domain policy or rule, leading to repeated plausible but incorrect fixes in later iterations. The authors propose the method of agentic synthesis against counterexample-supplemented sketches (Sketch-CE), a repository-native, human-in-the-loop approach that integrates explicit operator approval for counterexamples exposing missing or mistaken policy. A partial, code-shaped sketch evolves with each approved counterexample, guiding code and prompt generation and preserving provenance across an accepted archive and curated regression set. The method emphasizes maintaining explicit connections between counterexamples, policy revisions, and executable checks, ensuring that accepted corrections become durable and inspectable governing policy rather than ephemeral repair attempts.

Their implementation, CatSynth, applies this process to a synthetic browser application with a coding agent (GPT-5.4-mini). A single open-world run showed that evolving the sketch via operator-approved counterexamples improved generalization to unseen cases (passing 19 of 21 withheld vs 15 of 21 from the initial sketch with replay-all). Retaining code and minimizing repair calls reduced artifact churn notably (719 lines over 9 calls vs 2,394 lines over 15 calls). This provides tangible evidence that the evolved sketch captures reviewed policy and that this repository-native workflow can enforce policy durability without reliance on accumulated prompt context or informal rules in conversation memory. However, the authors acknowledge the evaluation is limited to one model and reveal order and thus only establishes the validity of the approach under these conditions rather than broad superiority or correctness.

Key findings

• In an open-world CatSynth run using GPT-5.4-mini, 8 of 14 frozen candidate cases became operator-approved counterexamples.
• Rebuilding from the evolved sketch passed 19 of 21 withheld test cases compared with 15 of 21 when rebuilding from the initial sketch and replaying all accepted examples.
• Retaining code across counterexamples required 9 developer calls and produced 719 lines of cumulative artifact churn versus 15 calls and 2,394 lines for the replay-all approach.
• All three agentic synthesis paths passed the 8 accepted counterexamples after promotion to the regression set.
• The gate mechanism uses replay plus semantic comparison over a curated regression set to detect regressions and policy violations, distinguishing between state repair and policy preservation.
• Every accepted counterexample mandates a sketch revision reviewed by an operator, ensuring that policy changes are explicit and durable.
• Oracle A (deterministic code) and Oracle B (prompt-mediated completion) enable handling both formalized and narrative domain rules under the same gating and sketch discipline.
• Periodic clean regeneration from the evolved sketch and known-code anchors validates that learned policy is captured in the sketch itself rather than fuzzy prompt histories.

Threat model

The paper addresses adversarial coding agents that generate plausible but incorrect repairs violating unencoded domain rules. The adversary in this context is the autonomous repair process that might silently circumvent SME policies due to lack of encoded constraints. The adversary cannot unilaterally change operator-approved policies or the curated regression set, as explicit human approval is required for policy changes. The system relies on human-in-the-loop governance to prevent unchecked policy degradation.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is implicit: coding agents (such as LLMs) that autonomously generate repairs to failing cases can introduce plausible but incorrect fixes that do not preserve the underlying domain policy. The model assumes the presence of human subject-matter experts (SMEs) who recognize policy violations and operators who explicitly approve policy changes. The coding agent does not decide policy truth; revisions to policy-bearing artifacts require explicit operator sign-off.

2. Data: CatSynth, a synthetic browser application with public domain rules, serves as the experimental dataset. It consists of frozen candidate failure cases, input/output corrections, and operator-approved counterexamples. Regression sets are curated subsets of accepted counterexamples that serve as regression tests. The archive preserves full provenance, linking failures, corrected outputs, rejecting repairs, and sketch changes.

3. Architecture / Algorithm: The system revolves around the counterexample-supplemented sketch (Sketch-CE), a partial code-shaped sketch encoding current domain policy through prose, tables, pseudo-code, and examples. The Developer (coding agent) generates deterministic code (Oracle A) and prompt text (Oracle B) to implement the policy. The gate is composed of replay and semantic comparison checks verifying state repair and policy consistency on the regression set. Operator-approved counterexamples update the sketch and archive, enforcing an explicit policy evolution cycle.

4. Training Regime & Pipeline: While no training is involved per se, the method iteratively proceeds as follows: starting from an initial sketch and empty implementation, a candidate case failing the current policy surface is proposed with correction and policy explanation by the SME. The operator approves the counterexample, requiring a sketch revision. The Developer edits sketch, code, and prompts under known-code anchors and current policy. The gate validates that the new implementation passes the active case plus regression cases before the next counterexample is exposed. Periodic clean regeneration discards implementation and recompiles from sketch and anchors alone to verify policy capture.

5. Evaluation Protocol: Evaluation metrics include the number of accepted counterexamples, regression set sizes, pass/fail counts on withheld cases, developer call counts, and lines of code changed. The authors compared rebuilding from the initial sketch with replay-all accepted counterexamples versus rebuilding from the evolved sketch with retained implementation. The gate’s replay and semantic comparison logs help distinguish state repair from policy adherence. Only one model (GPT-5.4-mini) and one counterexample reveal order was evaluated.

6. Reproducibility: CatSynth is a synthetic artifact with preserved fixtures, tests, provenance, and audit data included in the appendix. The authors provide a captured comparison and audit trail enabling inspection of the process but no mention of public code release or frozen model weights. The dataset is synthetic and publicly representational rather than real production data.

Example end-to-end: An operator receives a failing candidate case with a corrected expected output and policy explanation from the SME. The candidate is approved as a counterexample only if it reveals a missing or mistaken policy in the current sketch. The Developer integrates this policy change by revising the sketch, deterministic code, and prompt surfaces under defined known-code anchors. The gate is run with this active case plus the regression set to check for regressions. If regressions occur, the Developer repairs implementation iteratively until the gate passes. Once the gate is green, the counterexample is included in the regression set, and the next candidate failure can be processed. Periodically, implementation is reset and regenerated from the evolved sketch to confirm policy retention without relying on prompt history or patch replay.

Technical innovations

• The formulation of a repository-native artifact called the counterexample-supplemented sketch that explicitly encodes learned policies discovered through operator-approved counterexamples.
• A repair loop that integrates explicit operator counterexample approval with enforced sketch revisions, preventing silent or unreviewed policy drift common in purely patch-based coding agent workflows.
• Dual-oracle implementation surfaces splitting deterministic code (Oracle A) and prompt-mediated narrative completion (Oracle B) under unified gating and sketch discipline.
• A regression gate applying replay plus semantic comparison over a curated subset of operator-approved counterexamples to enforce policy preservation with bounded claims rather than full formal proofs.

Datasets

• CatSynth — synthetic dataset of 14 frozen candidate test cases with operator-approved counterexamples and a curated regression set — publicly derived synthetic application

Baselines vs proposed

• Rebuild from initial sketch with replay-all accepted examples: pass 15 of 21 withheld cases
• Rebuild from evolved sketch after counterexample-driven revisions: pass 19 of 21 withheld cases
• Retain code across counterexamples: 9 developer calls, 719 lines churn, pass 18 of 21 withheld cases
• Replay-all approach: 15 developer calls, 2,394 lines churn

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.15854.

Fig 1 (page 16).

Fig 2 (page 25).

Fig 3 (page 30).

Fig 4 (page 31).

Limitations

• Evaluation is based on a single coding agent model (GPT-5.4-mini) and a single counterexample reveal order; generality remains unproven.
• The policy boundaries and regressions are curated by maintainers and operators, making results sensitive to human decisions and review rigor.
• The method assumes the presence of qualified SMEs and operators, which may limit applicability in fully automated or less expert environments.
• Formal correctness is not claimed; the gate provides a bounded check limited to the regression set which is necessarily incomplete.
• Narrative policies handled by prompt-mediated completions remain partly under-specified and depend on model behavior constrained but not fully controlled by gating.
• The approach currently focuses on repository-native code and prompt repair without addressing adversarial or malicious inputs.

Open questions / follow-ons

• How well does the Sketch-CE approach scale to larger, real-world codebases with complex, evolving policies?
• Can the process be extended to multiple coding agents working concurrently with conflicting revisions or distributed teams of operators?
• What techniques could increase automation in operator approval or SME policy extraction without sacrificing review rigor?
• How does the approach interact with adversarial or deceptive model behavior aiming to bypass review or patch policies?

Why it matters for bot defense

For bot-defense or CAPTCHA practitioners, this work exemplifies a rigorous human-in-the-loop methodology to evolve and preserve domain-specific policy constraints in code and prompt-driven implementations used in automated systems. The explicit approval and sketch revision process offers a reproducible and inspectable approach to embed evolving security or usability policies into agent-generated code artifacts safely, avoiding silent policy drifts that could lead to exploit or circumvention. Maintaining a curated regression set and semantic gate could analogously support continuous validation of bot detection algorithms or challenge-response logic as policies adapt, enforcing durable, reviewable policy artifacts.

However, the method relies on human judgment to approve policy changes, so scaling or automating governance remains a critical challenge. Bot-defense engineers might consider adopting similar provenance and artifact linkage discipline when managing complex adaptations of automated challenge generation or solver mechanisms in evolving threat environments.

Cite

@article{arxiv2607_15854,
  title={ Agentic Synthesis against Counterexample-Supplemented Sketches },
  author={ Muness Castle and Eric Rubeck },
  journal={arXiv preprint arXiv:2607.15854},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.15854}
}

Read the full paper

• arXiv:2607.15854 (abstract)
• arXiv:2607.15854 (PDF)