PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks
Source: arXiv:2607.28587 · Published 2026-07-30 · By Manyi Wang, Junjielong Xu, Pinjia He
TL;DR
SWE-bench-like benchmarks are widely adopted for evaluating Large Language Models (LLMs) on issue resolution tasks, pairing real-world GitHub issues and pull requests (PRs) as problem statements and oracles. However, this construction assumes perfect alignment between the PR and its linked issue, an assumption that frequently breaks down due to development complexities. The paper conducts a systematic empirical study on SWE-bench Verified, identifying that 13.6% of PR-Issue pairs exhibit misalignment across five distinct patterns, such as scope creep, defects, incomplete specs, and follow-up PRs. Misaligned instances correlate strongly with difficulty and failure rates in LLM agent evaluations, indicating distorted benchmark reliability. To address this, the authors propose PaiChecker, a three-phase multi-agent framework that first applies pattern-specific textual analysis via dedicated subagents, then synthesizes and self-corrects labels, and finally verifies alignment at code-level. Experiments on two SWE-bench-like datasets and four LLM backbones demonstrate PaiChecker significantly outperforms prompting and agent baselines, achieving up to 92.12% binary accuracy. The work highlights the critical need for careful benchmark curation and the benefits of a nuanced, multi-modal detection approach.
Key findings
- 13.6% (68 out of 500) of SWE-bench Verified instances have PR-Issue misalignment.
- Misalignment spans five high-level patterns and 11 fine-grained scenarios, including PR Scope Creep (32.4%), Defective PR (44.1%), and Incomplete Specification (26.5%).
- 41.2% of instances never resolved by any of 131 leaderboard agents are misaligned, compared to only 5.7% misalignment among instances solved by more than 100 agents.
- Excluding misaligned instances raises agent pass rates by an average of 2.77 percentage points and changes ranking for 64.1% of agents, including 9 of the top 10 positions.
- PaiChecker achieves 92.12% binary accuracy and 84.66% exact match on SWE-Gym, outperforming the strongest baseline by 5.13–12.39 accuracy points and 9.02–17.76 exact-match points.
- On SWE-bench Multilingual, PaiChecker achieves 91.67% binary accuracy and outperforms baselines by 2.33–5.67 accuracy points and 3.66–8.00 exact-match points.
- Ablation studies show each PaiChecker phase (pattern-specific subagents, label synthesis, code validation) contributes to performance gains.
- Monolithic prompting approaches fail to generalize beyond predefined patterns and produce inconsistent labels due to heterogeneous reasoning needs and artifact subsets.
Threat model
The study assumes an adversary is the inherent noise and errors in benchmark construction pipelines that cause mismatches between PRs and issues, rather than an active attacker trying to evade detection. The adversary cannot alter the benchmark post-hoc but causes distorted evaluations through mislabeling. The defender attempts to detect such misaligned pairs to improve benchmark reliability.
Methodology — deep read
The paper focuses on detecting misalignment between PRs and linked issues in SWE-bench-like benchmarks, where misalignment means discrepancy or inconsistency in scope, completeness, or correctness. The threat model assumes a benchmark curator trying to identify erroneous PR-Issue pairs; the adversary is the implicit noise or errors in the construction pipeline, rather than an active attacker.
The main dataset for empirical study is SWE-bench Verified, a human-validated subset of 500 instances from SWE-Bench. Each instance includes: issue-side data (description, discussion), PR-side data (description, discussion, commits, changed files, code), and metadata linking them. Manual open coding was applied to analyze all instances for misalignment patterns and prevalence.
PaiChecker’s core architecture consists of a three-phase multi-agent system adhering to a text-driven, code-validation principle:
Phase I: Pattern-Specific Identification with three specialized subagents (Issue Analyzer, PR Scope Analyzer, PR Connection Analyzer). Each subagent focuses on a subset of misalignment patterns by analyzing corresponding artifacts: Issue Analyzer compares issue description vs. discussion for incomplete specs; PR Scope Analyzer compares issue description vs. PR description and patch to detect scope creep and unspecified literals; PR Connection Analyzer traces temporal cross-PR dependencies using GitHub API to detect defective or follow-up PRs.
Each subagent produces structured judgements, evidence, and suspicious clues for beyond-taxonomy signals.
Phase II: Label Synthesis and Textual Self-Correction combines subagent outputs via a coordinating agent to finalize preliminary labels, reject contradictory labels, and assign an 'Others' category for beyond-taxonomy cases. This phase addresses monolithic approach failures by aggregating cross-subagent evidence and ensuring textual consistency between labels and reasoning.
Phase III: Code Validation performs code-level verification of Phase II’s textual label via cross-referencing added/modified code constructs, test patches, commit history, and auxiliary GitHub context. This step catches false positives where textual claims fail to reflect actual code changes.
The three-phase design balances specialization, self-correction, and multi-modal evidence for more precise detection.
For example, in a misaligned instance where a PR was accidentally linked to an unrelated issue (Figure 4), the PR Scope Analyzer initially detected scope creep, but the Coordinator reconciled evidence across subagents and vetoed incorrect labels, while code validation confirmed the actual functionality.
Training and evaluation involved four LLM backbones (GPT-5.3 Codex, Qwen-3.5 Plus, Gemini-3.1-Pro Preview, Claude-Sonnet-4.6) tested on two SWE-bench-like datasets: SWE-Gym and SWE-bench Multilingual. Baselines included chain-of-thought prompting, Mini-SWE-Agent, and other agent frameworks.
Metrics included binary alignment accuracy and exact match on final labels. Ablations studied the impact of each phase.
The study also collected official leaderboard resolution counts for 131 agents on SWE-bench Verified, enabling correlation analysis between misalignment and agent resolution rates.
Code and annotated data were released to foster future benchmark curation.
Reproducibility is supported by detailed phase descriptions and released artifacts, though some GitHub API dependency implies potential variability in live runs.
Technical innovations
- A multi-agent three-phased architecture decomposing heterogeneous misalignment detection into focused subagents for distinct pattern groups (Issue Analyzer, PR Scope Analyzer, PR Connection Analyzer).
- A coordinating agent synthesizes subagent outputs and suspicious clues to enable beyond-taxonomy label assignment and ensures textual reasoning-label consistency via self-correction.
- A code-level validation phase cross-references textual claims against actual code diffs, test patches, and repository context to verify and correct textual judgments, implementing a two-level self-correction mechanism.
- The text-driven, code-validation principle leverages natural language artifacts (issue and PR descriptions, discussions) first for semantic alignment detection, then validates with concrete code evidence, effectively exploiting complementary signal modalities.
Datasets
- SWE-bench Verified — 500 instances — public human-validated subset of SWE-Bench with issue-PR pairs and leaderboard resolution data
- SWE-Gym — size not specified in excerpt — public SWE-bench-like benchmark
- SWE-bench Multilingual — size not specified in excerpt — public SWE-bench-like benchmark
Baselines vs proposed
- Mini-SWE-Agent prompting baseline on SWE-Gym: binary accuracy range ~79%–87% vs PaiChecker: up to 92.12%
- Chain-of-Thought (CoT) prompting on SWE-Gym: lower accuracy and consistency than PaiChecker
- Other agent framework baselines on SWE-bench Multilingual: binary accuracy ~86%–89% vs PaiChecker: up to 91.67%
- Exact match scores on SWE-Gym: baselines 67–75% vs PaiChecker 84.66%
- Exact match scores on SWE-bench Multilingual: baselines 74–81% vs PaiChecker up to 85%
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.28587.
Fig 5: Workflow of PaiChecker. Phase I performs specific misalignment identification through three specialized subagents.
Fig 4: Motivating Example iterative__dvc-4872: PR #4872 is accidentally linked to issue #4796 but actually fixes issue #4504.
Fig 3 (page 6).
Fig 4 (page 6).
Fig 5 (page 6).
Fig 6 (page 6).
Limitations
- Manual coding for taxonomy derivation performed on a limited (500) instance subset; broader SWE-bench-like datasets may reveal additional patterns.
- The reliance on public GitHub API for PR Connection Analyzer introduces variability and potential rate-limit issues affecting detection consistency.
- Evaluation focuses on binary and exact label accuracy; real-world impact on downstream LLM training or evaluation metrics beyond leaderboard pass rates remains unquantified.
- No adversarial robustness testing: the approach assumes honest, if noisy, benchmark constructions rather than adversarially manipulated PR-Issue pairs.
- The generalizability of PaiChecker beyond SWE-bench-like benchmarks to other code-issue datasets or domains is not demonstrated.
- Dependency on high-quality natural language artifacts (issue and PR discussions) may limit applicability to repositories with sparse or informal discourse.
Open questions / follow-ons
- How does PR-Issue misalignment affect the training efficacy and generalization of code generation models beyond evaluation metrics?
- Can PaiChecker’s multi-agent framework be extended to detect misalignment in different software development ecosystems or private repositories with limited metadata?
- What is the impact of integrating dynamic behavioral analysis rather than only static code validation for confirming alignment?
- How robust is PaiChecker against intentional adversarial manipulation (e.g., obfuscated or misleading PR and issue descriptions)?
Why it matters for bot defense
For bot-defense engineers and CAPTCHA practitioners developing LLM-based code generation or automated repair agents, this work underscores a critical data quality issue: the ground-truth mappings in issue-resolution benchmarks can be substantially noisy due to PR-Issue misalignments. Reliance on such benchmarks for training or evaluation can lead to misleading conclusions about agent capabilities and unfair scoring. PaiChecker’s approach demonstrates that automated, multi-modal detection combining textual reasoning with code validation can help curate cleaner, more reliable benchmark datasets, ultimately improving agent training signals and evaluation fairness. Integrating such misalignment detection could improve the robustness and interpretability of LLM-based software agents used in automated workflows that may interface with security-critical or adversarial environments.
Cite
@article{arxiv2607_28587,
title={ PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks },
author={ Manyi Wang and Junjielong Xu and Pinjia He },
journal={arXiv preprint arXiv:2607.28587},
year={ 2026 },
url={https://arxiv.org/abs/2607.28587}
}