How Humans, Bots, and Agents Communicate About Vulnerabilities in Pull Requests
Source: arXiv:2606.28125 · Published 2026-06-26 · By Pien Rooijendijk, Christoph Treude, Mairieli Wessel
TL;DR
This study investigates how vulnerabilities are communicated in GitHub pull requests by humans, bots, and coding agents, focusing on both explicit vulnerability identifiers (e.g., CVE, GHSA) and implicit security-related language (e.g., "unauthorized access," "SQL injection"). Prior research mainly analyzed explicit signals, potentially missing substantial security discussions conveyed through natural language. The authors build on the AIDev-pop dataset, which contains over 33,000 pull requests authored by coding agents, and extend it with additional human and bot pull requests from the same repositories, enabling a comprehensive comparison of vulnerability communication across actor types and pull request components.
The methodology involves detecting explicit vulnerability references via regexes matching standard identifiers and implicit signals using a validated keyword- and context-based approach. These signals are extracted from multiple pull request artifacts including titles, descriptions, review comments, commit messages, and timeline discussions. The study further cross-validates these references using static analysis (via Semgrep) to determine whether the referenced vulnerabilities correspond to actual code changes that introduce or fix security issues. Lastly, the study analyzes how vulnerability references influence social review dynamics and pull request outcomes.
Overall, this work contributes a novel large-scale empirical analysis detailing how humans, bots, and coding agents discuss vulnerabilities explicitly and implicitly in pull requests, and connects communication practices to actual security impact and social processes in developer collaboration.
Key findings
- The AIDev-pop dataset contains 33,078 pull requests authored by 5 coding agents across 2,807 popular repos (>500 stars).
- Explicit vulnerability references are detected using regex patterns for standardized IDs (CVE, CWE, GHSA, etc.) in pull request artifacts such as titles, descriptions, comments, and commits.
- Implicit security-related signals are extracted from natural-language security keywords combined with contextual filtering adapted from Zhou and Sharma's method.
- Both explicit and implicit vulnerability references appear in various pull request components; their distribution varies significantly by actor type (human, bot, coding agent) and artifact.
- Static analysis with Semgrep on pull request code compares base and head commits to categorize vulnerability findings as introduced, fixed, or unchanged.
- Regression models analyze associations between reference types and actual vulnerability changes while controlling for pull request size, complexity, and repository effects.
- Social response analysis shows that pull requests with explicit vulnerability references tend to receive more review comments, more reviewers, faster response times, and higher merge rates than implicit or no reference cases.
- The study validates signal detection with LLM-assisted annotation (Gemini 2.0) and manual reviews, reporting Cohen's kappa inter-rater reliability and classification accuracy.
Threat model
The study does not explicitly specify an adversarial threat model. However, the implicit assumption involves developers, bots, and coding agents communicating vulnerabilities, possibly in adversarial settings. Adversaries may submit malicious pull requests introducing vulnerabilities without explicit references. Bots and agents are treated as distinct actors with possibly different motivations. The adversary capabilities are not formally defined, and the study concentrates on understanding communication practices rather than defense mechanisms.
Methodology — deep read
The study is designed as a large-scale empirical investigation on vulnerability references in GitHub pull requests authored by humans, bots, and coding agents. The threat model implicitly assumes adversaries could be malicious actors indirectly affecting development practices but is not a formal adversarial study.
Data provenance centers on the AIDev-pop dataset, which filters the AIDev dataset to include PRs from popular repositories (stars >500) with 33,078 agent-authored PRs across 2,807 repos. To extend coverage, all PRs during the same timeframe from these repos are collected, allowing comparison across actor types.
Explicit vulnerability references are detected with curated regexes targeting standard identifier patterns from vulnerability databases (e.g., CVE-####-####, GHSA-xxxx-xxxx-xxxx). Detection is done independently on PR titles, descriptions, review comments, commit messages, and timeline events. Each detection is linked to the artifact and author account.
Implicit vulnerability signals are detected via a keyword-based approach following prior work by Zhou and Sharma (2017), which combines security-related keywords with contextual filtering to distinguish true security discussion from generic bug-fixing language. The same artifact types as explicit are scanned.
Detected signals undergo validation using the Gemini-2.0-flash large language model to classify whether a detected reference truly corresponds to a vulnerability-related communication. Two human annotators independently label stratified random samples of 360 PRs for explicit and implicit signals respectively. Cohen’s kappa measures inter-rater agreement. Disagreements are reconciled to form ground truth for evaluating LLM labeling performance.
Authors classify reviewer accounts into coding agents (based on AIDev-pop labels), bots (based on a combination of GitHub's bot tags, username heuristics, and a curated bots list), and humans (everything else). Manual verification of samples ensures classification reliability.
To evaluate correspondence of vulnerability references to actual code changes, the Semgrep static analysis tool is applied to all commits in each PR. The base commit (pre-change) and head commit (post-change) are compared to determine if vulnerabilities were introduced, fixed, or unchanged. Findings are matched by vulnerability ID and file path to track specific issues across intermediate commits if present.
For social response analysis, metrics such as number of review comments, number of reviewers, merge status, and time to first response are collected and compared between PRs with explicit and implicit vulnerability references, controlling for PR characteristics (lines changed, files changed, etc.) and repository level clustering.
Statistical analyses include chi-square tests, regression models (logistic, count models), and time-to-event analyses to probe relationships between reference types, actor types, vulnerability outcomes, and social activity.
As a concrete example, a PR authored by a coding agent containing an explicit CVE identifier in the description is scanned with Semgrep to check if the CVE issue is addressed in the changed lines. Review comments and merge status are then analyzed to see how the explicit reference influenced the PR's evaluation compared to PRs with implicit or no references.
Code and dataset availability is not explicitly mentioned; AIDev-pop is a referenced prior dataset and Semgrep is open source, suggesting reproducibility but dependent on external datasets and tools.
Technical innovations
- Combining explicit vulnerability identifier detection with contextual implicit security-related signal detection across multiple pull request artifact types.
- Linking vulnerability references from pull requests to Semgrep static analysis results on PR code to assess whether references correspond to actual introduced or fixed vulnerabilities.
- Comprehensive actor classification of pull request authors into humans, bots, and multiple coding agents to analyze communication differences.
- Application of LLM-assisted annotation combined with manual validation to reliably identify true vulnerability communications from noisy textual signals.
- Integrative analysis connecting vulnerability reference types with social pull request metrics such as review activity, response time, and merge outcomes.
Datasets
- AIDev-pop — 33,078 pull requests — subset of AIDev dataset, containing coding agent authored PRs from 2,807 popular GitHub repositories
- Extended human and bot pull requests collected from the same repositories as AIDev-pop during the same observation period
Baselines vs proposed
- Explicit vulnerability references vs implicit signals: Explicit references appear more frequently in certain pull request components and are more strongly associated with actual vulnerability fixes as detected by Semgrep (detailed numbers not yet reported).
- Pull requests with explicit vulnerability references receive statistically significantly more comments, more reviewers, faster first response times, and higher merge rates compared to those with only implicit references or no references, controlling for PR size and repository.
- Coding agents vs bots vs humans: The distribution of explicit and implicit vulnerability references varies significantly across these actor types (statistical test details unspecified).
- LLM-assisted annotation accuracy assessed against human coding on stratified random samples shows high agreement but exact metrics are pending final manuscript.
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.28125.
Fig 1: Examples of explicit and implicit references & signals in pull requests.
Fig 2: provides an overview of the study design.
Fig 3 (page 4).
Fig 4 (page 6).
Fig 5 (page 6).
Limitations
- Detection of implicit vulnerability references relies on a heuristic keyword and context filtering approach which may result in false positives or false negatives despite validation.
- Static analysis with Semgrep may miss context-dependent or complex vulnerabilities, and may yield false positives.
- Account classification into bots, humans, and coding agents may have errors due to heuristics and incomplete bot lists.
- Data is limited to popular public GitHub repositories (>500 stars) and therefore may not generalize to private, enterprise, smaller, or non-GitHub projects.
- Focus on pull request artifacts excludes security communications occurring in external venues such as mailing lists, private disclosures, or issue trackers outside GitHub.
- The study is observational and correlational, so causation between vulnerability references and PR outcomes cannot be firmly established.
Open questions / follow-ons
- How often do vulnerability references (explicit or implicit) in pull requests correspond to incorrect or unsupported security claims, especially in AI-generated PRs?
- What are the underlying reasons for differences in vulnerability communication styles between humans, bots, and coding agents?
- How do vulnerability communication practices vary in private or enterprise repositories and non-GitHub ecosystems?
- Can vulnerability detection and communication be improved through contextual, multi-modal approaches integrating code, text, and social signals?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, this work highlights the nuanced ways vulnerabilities are communicated by different actor types in pull requests, including implicit language that might evade simple pattern matching. Understanding these communication patterns can inform the design of tools that detect malicious or automated behavior related to security disclosures within collaborative code review workflows. The distinction between explicit and implicit signals could help refine detection criteria for bots or agents generating security-related pull requests, enabling more targeted challenge mechanisms or automated triage.
Moreover, by linking textual signals to actual code changes verified with static analysis, this study provides an empirical basis to assess the reliability of security claims made by various actors. This connection is critical for designing defenses that prevent automated systems from manipulating security reviews via misleading or spurious vulnerability references. The social response analysis also implies that responses to security signals differ based on their explicitness, which bot-defense could leverage to prioritize human review or gating steps accordingly.
Cite
@article{arxiv2606_28125,
title={ How Humans, Bots, and Agents Communicate About Vulnerabilities in Pull Requests },
author={ Pien Rooijendijk and Christoph Treude and Mairieli Wessel },
journal={arXiv preprint arXiv:2606.28125},
year={ 2026 },
url={https://arxiv.org/abs/2606.28125}
}