MineValiCoder: Reliable Code Generation with Test Case Quality Mining and Bipartite Graph-Based Mutual Validation

Source: arXiv:2607.22471 · Published 2026-07-24 · By Zhen Zhao, Qihang Yang, Feifei Dai, Xiangfang Li, Bo Li

TL;DR

MineValiCoder addresses critical challenges in automated code generation via test-driven development (TDD) using Large Language Models (LLMs). Existing LLM-based TDD approaches depend heavily on human-crafted test cases or generate test cases of mixed quality, leading to misleading optimization feedback and unreliable code selection. MineValiCoder's novelty lies in a closed-loop framework that jointly mines high-quality test cases, iteratively refines code candidates in parallel, and applies bipartite graph-based mutual validation to reliably score and select the best code. This collaborative interaction between test case quality and code quality helps overcome LLM stochasticity and noisy testing signals.

Empirically, MineValiCoder significantly outperforms prior state-of-the-art techniques across multiple benchmarks (HumanEval, MBPP, APPS, LiveCodeBench) and backbone LLMs (GPT-4, Qwen, Llama3). It achieves Pass@1 scores of 96.34% on HumanEval, 87.40% on MBPP, 64.00% on APPS, and 51.33% on LiveCodeBench, demonstrating robust improvements in fully automated TDD with no reliance on human-crafted tests. The framework's modular design—including Test Case Quality Mining, Parallel TDD Refinement, and Bipartite Graph-Based Code-Test Mutual Validation—forms a novel, effective pipeline for reliable automated program synthesis.

Key findings

• MineValiCoder obtains Pass@1 scores of 96.34% on HumanEval, 87.40% on MBPP, 64.00% on APPS, and 51.33% on LiveCodeBench benchmarks.
• The Test Case Quality Mining (TCQM) module filters out faulty test cases by re-predicting expected outputs and retaining only cases with self-consistent outputs.
• Parallel TDD refinement runs N=8 independent code optimization pipelines, each guided by a distinct validated test suite, generating diverse high-quality code candidates.
• Monotonicity constraints ensure that newly optimized code fixes failed test cases without invalidating previously passed ones, which stabilizes iterative improvement.
• The Bipartite Graph-Based Code-Test Mutual Validation (BiCoTeV) module models code candidates and test cases as two sets of nodes linked by pass/fail edges, dynamically propagating reliability scores.
• A consistency-based filtering in BiCoTeV removes code candidates suspected of overfitting by comparing scores of passed and failed test cases, preventing selection bias from noisy tests.
• MineValiCoder outperforms baselines including Chain-of-Thought prompting, Self-Planning, Reflexion, CodeT, and MapCoder across four different large language models.
• Qwen3-4B, a relatively small LLM, is used to generate test cases in TCQM, demonstrating the framework's ability to mine reliable supervision signals from lightweight models.

Threat model

The threat model focuses on the inherent stochasticity and probabilistic errors of large language models used both for code generation and automatic test case generation. The adversary is essentially the noise and faults introduced by LLM outputs, causing faulty test cases that mislead code repair and noisy signals that confound code selection. The framework assumes no malicious tampering but seeks to robustly mitigate the unreliable feedback loops and ambiguous evaluation caused by imperfect LLM-generated tests.

Methodology — deep read

1. Threat Model and Assumptions: The adversary is the inherent stochasticity and noisiness of large language models (LLMs) in generating both code and test cases. Faulty or low-quality generated test cases mislead code optimization, and residual erroneous test cases cause conflicting evaluation signals, making reliable code selection difficult. The framework assumes no access to human-crafted test cases and relies solely on natural language problem descriptions. Adversaries cannot directly tamper with the code or tests but benefit from the inherent randomness and occasional hallucinations of LLM outputs.

2. Data: The evaluations use public benchmarks for code generation including HumanEval, MBPP, APPS, and LiveCodeBench. These datasets contain problem descriptions and reference test cases (used only for evaluation, not for training). MineValiCoder generates test cases automatically from natural language problem descriptions. The parallel sampling size N=8 is used for test suite and code candidate generation. Pass@1 is the primary metric, measuring the fraction of problems for which the top code candidate passes all test cases.

3. Architecture/Algorithm: MineValiCoder consists of three core modules: (a) Test Case Quality Mining (TCQM): Uses LLMs to generate diverse raw test cases for the task by prompting with instructions to cover normal, boundary, and edge scenarios. Then applies a self-validation mining step, where for each raw test case (input, output), the output is masked and re-predicted by the LLM given the input and problem description. Only test cases whose re-predicted output matches the original are retained, filtering out faulty test cases. (b) Parallel TDD Refinement: For each of the N validated test suites, an independent iterative optimization process is spawned. Starting from initial code generated from the problem, it executes against the test suite to identify pass/fail tests. Iteratively, for each failing test case, a 'Plan-then-Code' prompting mechanism is used, where the LLM first plans a fix based on the failing test, then generates updated code code. A strict monotonicity constraint requires that each update fixes the targeted failing test without breaking any previously passed ones, enforcing stable progress. If a test case remains unfixable after Rmax retries, it is temporarily marked to avoid infinite loops. (c) Bipartite Graph-Based Code-Test Mutual Validation (BiCoTeV): After generating multiple code candidates and collecting all unique validated test cases, this module constructs a bipartite graph with codes and test cases as two disjoint node sets. Edges represent which code passes which test. Scores for code and test nodes are updated iteratively, propagating credibility bidirectionally with damping factor d. This dynamic mutual validation converges to reliability scores reflecting the joint quality of code and tests. Additionally, a consistency-based filter removes code candidates that perform better on low-score failed tests than on high-score passed tests, identifying overfitting. The highest scoring candidate in the filtered set is selected as the final output.

4. Training Regime: The methodology focuses on inference-time generation; no supervised training or weight updates are performed. LLM sampling temperature for initial code generation is varied (0.8, 1.2, 1.5) to encourage diversity. For test generation and validation, Qwen3-4B is used with nucleus sampling (top_p=0.9), temperature 0.6 for stable code optimization. Each parallel refinement runs independently.

5. Evaluation Protocol: Pass@1 is measured as the primary metric. Benchmark tasks cover varying difficulty levels from trivial to competition programming problems. Comparisons are made against baselines including direct prompting, Chain-of-Thought, Self-Planning, Reflexion, CodeT, and MapCoder, across multiple LLMs (GPT-4, Llama 3.1 8B, Qwen 2.5-Coder 7B, Qwen 3 4B). Ablations explore the impact of TCQM filtering, monotonicity constraints, and BiCoTeV mutual validation. Multiple test suites and code candidates are sampled (N=8) to ensure results' statistical robustness.

6. Reproducibility: The paper provides a link to anonymized code and instructions for replication. The benchmarks used are publicly accessible. The methodology relies on prompting publicly available LLMs or open-source models like Qwen and Llama3. The iterative refinement pseudocode is thoroughly described. However, some details such as exact model checkpoints, seeds, and specific hyperparameter values for optimization retries Rmax are not fully detailed in the excerpts.

Concrete example: For a given function to generate (e.g., square(x)), MineValiCoder first generates multiple test suites containing inputs and outputs via TCQM and validates them by re-prediction. Each test suite drives an independent TDD pipeline where initial code (return x) is iteratively refined to pass currently failing tests without breaking passed ones, using 'Plan-then-Code' prompting. Multiple different code candidates emerge (return x2, return x*3, return x+2). These candidates and test cases form a bipartite graph, scored mutually by BiCoTeV. Candidates passing reliable test cases receive higher scores. Overfitting codes that pass erroneous tests but fail reliable ones are filtered out. The stable highest scoring correct code (return x2) is selected as the final solution.

Technical innovations

• Test Case Quality Mining (TCQM) leverages LLM self-consistency to filter out faulty generated test cases by masking and re-predicting outputs, ensuring reliable optimization supervision.
• Parallel TDD Refine runs N independent iterative code repair pipelines with monotonicity constraints preventing regression of passed test cases, producing diverse, high-quality candidate code.
• Bipartite Graph-Based Code-Test Mutual Validation (BiCoTeV) models code-test passing relations as a bipartite graph and uses dynamic score propagation to jointly assess and select the optimal code under noisy test conditions.
• Consistency-based filtering within BiCoTeV identifies and removes overfitting candidates that perform better on less reliable failed tests than on more reliable passed tests.

Datasets

• HumanEval — 164 problems — public GitHub dataset for code generation benchmark
• MBPP (Mostly Basic Programming Problems) — 974 problems — public dataset
• APPS (Automated Programming Progress Standard) — 150 problems sampled from Introductory, Interview, and Competition levels — public dataset
• LiveCodeBench — 150 randomly sampled problems — public dataset

Baselines vs proposed

• Direct Prompting: Pass@1 = varies, MineValiCoder outperforms significantly (e.g., 96.34% vs ~70% on HumanEval, exact baseline values not specified)
• Chain-of-Thought (CoT): inferior code generation quality compared to MineValiCoder by margins of 10+ percentage points on standard benchmarks
• Self-Planning: underperforms compared to MineValiCoder in code correctness measures across all benchmarks
• Reflexion: iterative self-debugging method lags behind MineValiCoder’s highest code correctness scores
• CodeT: majority voting over tests achieves lower Pass@1 than MineValiCoder’s mutual validation approach
• MapCoder: multi-agent programming lifecycle approach performs weaker than MineValiCoder’s test-quality-mining and graph mutual validation pipeline

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.22471.

Fig 1: Key challenges in automated TDD with LLM-generated

Fig 2: Overview of MineValiCoder. The framework first employs TCQM to generate validated test cases, followed by Parallel

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 4).

Limitations

• The method assumes availability of a sufficiently capable LLM for generating plausible test cases and reasoning; performance may degrade with weaker models.
• Residual noisy or erroneous test cases can remain after filtering; while BiCoTeV mitigates their impact, complete elimination of noisy tests is not guaranteed.
• The approach focuses on single-function or limited programming task settings; its scalability to large-scale multi-file projects or real-world software engineering is not studied.
• The evaluation does not explicitly report robustness to adversarial test cases crafted to mislead code optimization.
• The monotonicity constraint and multiple retries add computational overhead, potentially limiting real-time or resource-constrained scenarios.
• Training or fine-tuning was not part of the methodology; the framework depends entirely on inference-time prompting, potentially limiting adaptability to domain-specific programming languages.

Open questions / follow-ons

• How does MineValiCoder perform on larger, real-world software projects involving multi-module dependencies and integration testing?
• Can the proposed bipartite graph mutual validation concept be extended to other iterative AI generation tasks with noisy evaluation signals?
• What are the theoretical convergence properties and computational complexities of the dynamic mutual validation scoring algorithm?
• How robust is MineValiCoder to adversarially crafted or intentionally misleading test cases designed to subvert automated code optimization?

Why it matters for bot defense

MineValiCoder’s approach to closed-loop mutual validation between generated inputs (test cases) and outputs (code candidates) under noisy and stochastic generation offers insights applicable beyond code generation, including bot-defense systems reliant on challenge-response validation. In CAPTCHA or bot-defense contexts, dynamic mutual validation can help isolate reliable challenge components from noisy or adversarial inputs, improving the robustness of automated decision-making pipelines. Furthermore, the monotonicity-constrained iterative refinement and self-consistency validation of test cases suggest mechanisms to avoid misclassification from erroneous or sensor-faulty signals common in security systems. While MineValiCoder addresses software correctness, its principles can inspire more stable challenge generation and evaluation processes in automated bot-defense. However, direct application requires adaptation because CAPTCHA systems face active adversaries with different threat vectors rather than stochastic model errors.

Cite

@article{arxiv2607_22471,
  title={ MineValiCoder: Reliable Code Generation with Test Case Quality Mining and Bipartite Graph-Based Mutual Validation },
  author={ Zhen Zhao and Qihang Yang and Feifei Dai and Xiangfang Li and Bo Li },
  journal={arXiv preprint arXiv:2607.22471},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.22471}
}

Read the full paper

• arXiv:2607.22471 (abstract)
• arXiv:2607.22471 (PDF)