Detecting AI Coding Agents in Open Source: A Validated Multi-Method Census of 180 Million Repositories

Source: arXiv:2606.24429 · Published 2026-06-23 · By Arsham Khosravani, Audris Mockus

TL;DR

This paper addresses the problem of accurately detecting generative AI coding agents in the vast open-source software (OSS) ecosystem, where diverse and often invisible traces make prevalence estimates challenging. Prior censuses rely on single detection signals like bot accounts or configuration files, but the authors present a comprehensive multi-method detection framework classifying AI traces into four behavioral types (bot account, commit message signature, distributed human attribution, configuration file evidence). Applied over 180M+ Git repositories across multiple snapshots (Dec 2024–Apr 2026) using the World of Code infrastructure, this approach reveals a large undercount in single-channel measurements, with the union detecting up to 30× more activity for some agents such as Claude Code. Cross-validation with a separate pull-request-based dataset reveals near-disjoint agent populations and qualitatively different activity types, demonstrating that no single detection channel provides a representative census.

The multi-method approach exposed substantial silent adoption, bimodal adoption timing, and agent-specific contribution profiles, with commit-level agents like Claude Code focusing on maintenance work while cloud PR-based agents like Codex tend to deliver feature work. The work also shows AI agent activity growing rapidly, peaking at over 320K commits/month by early 2026. Validation with 495 hand-labeled samples and statistical confidence supports the precision of detection. Overall, the paper establishes a comprehensive, validated baseline census of AI coding agents’ presence and influence in open-source repositories, highlighting the complexity of this emerging software supply chain phenomenon.

Key findings

• Multi-method detection identifies 850,157 Claude Code commits in V2510 snapshot vs only 28,154 via bot-account lookup alone, a 30× relative recall gap (Finding 4).
• Commit-attributed AI agent activity exceeds 320,000 commits per month by April 2026 (V2604 snapshot).
• Configuration-file scanning (Type D) uncovers 21,078 Claude Code adopting projects with 888,177 distinct blob occurrences absent in December 2024 snapshot (Finding 2).
• The commit-based and pull-request-based (AIDev) censuses capture nearly disjoint populations; a PR census misses 79% of commit-detected Claude Code adopters and virtually all Codex adopters (Table 6).
• Replit dominates explicit commit message attributions with 385,668 commits (V2412), far exceeding other agents (Table 1).
• Agent work profiles differ by deployment mode: commit-channel agents (Claude Code, OpenHands, Aider) skew towards maintenance/bug fixes, while PR-channel agents (Codex, Cursor) skew towards feature work (Figure 3b).
• Adoption timing is bimodal across projects: agents are integrated either at project inception or after years of development ( Section 4.3).
• Configuration file conventions evolve rapidly: Claude Code emerges as dominant Type D agent in V2604 with nearly double the blob counts compared to earlier snapshots (Table 5).

Threat model

The adversary is not explicitly modeled as a malicious actor evading detection; rather, the focus is to identify AI coding agents embedded in open-source repositories that may leave varied and sometimes silent traces. It is assumed agents do not deliberately conceal or obfuscate all detectable metadata signals. The detection framework does not account for adversaries attempting to circumvent authorship or configuration-file patterns.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is not explicitly modeled as an attacker attempting to evade detection but the study focuses on reliably identifying AI coding agent traces embedded in open-source repositories. The assumption is that AI agents leave identifiable metadata signatures in version control history via bot accounts, commit messages, author name suffixes, or configuration files. There is no assumption that agents attempt to conceal or obfuscate these signals.

2. Data: The study uses three time-separated snapshots (V2412 Dec 2024, V2510 Oct 2025, V2604 Apr 2026) from the World of Code (WoC) infrastructure, indexing over 180 million Git repositories from multiple platforms (GitHub, GitLab, Bitbucket, and others). Data includes commit metadata, author IDs, file blobs, and project structures. Identity resolution infrastructure collapses 38 million author IDs into consolidated developer identities.

3. Architecture / Detection Framework: The core innovation is a multi-layer detection taxonomy (Types A-D):

• Type A: Centralized bot accounts identified via exact email lookups in author metadata (O(1) hash lookups).
• Type B: Commit-message signature detection via case-insensitive regex queries over commit messages stored in WoC's ClickHouse databases.
• Type C: Distributed human attribution where authors append tool-specific suffixes to author names (e.g., '(aider)'); detected via pattern scans of author-name shards.
• Type D: Configuration-file detection by scanning for presence of agent-specific config files (e.g., .replit, CLAUDE.md) via strict file-to-project blob mappings. These methods produce partially overlapping but complementary views that, when combined as a union, significantly increase recall.

4. Training / Rule Development: Detection rules (regexes, suffix lists, config file patterns) were curated and recalibrated as agent conventions evolve, e.g., adding new config files for Codex and Claude. 495 hand-labeled samples were used for validation with per-cell precision and Wilson confidence intervals.

5. Evaluation Protocol: Validation of detection precision via hand labeling; recall is lower bounded by union of methods but absolute recall is unknown. Prevalence and activity metrics reported over snapshots; comparison to external PR-based AIDev dataset to illustrate channel disjointness. No adversarial robustness tested.

6. Reproducibility: The World of Code infrastructure is a public platform for large-scale repository mining but some data snapshots and identity mappings may not be fully public. The exact detection code is not explicitly stated as released; validation methodology and audit trails are documented.

Example end-to-end detection for Claude Code in V2510: querying bot account emails yields 28,154 commits, message-signature regexes add 821,824, with intersection 21,971; union totals 850,157 commits showing 30× undercount if relying on bot account alone.

Technical innovations

• A four-tier AI coding agent detection taxonomy that integrates complementary signals (bot accounts, commit messages, author-name suffixes, and project configuration files) for comprehensive ecosystem-wide detection, contrasting with prior single-channel approaches.
• Application of multi-method detection on the World of Code infrastructure to enable O(1) hash lookups and scalable querying across 180 million repositories.
• Validated detection framework with 495 hand-labeled samples per detection-cell, employing Wilson confidence intervals to quantify precision rigorously.
• Demonstration of detection channel disjointness by cross-comparing commit-based detection with independent PR-based census (AIDev), revealing near-disjoint agent populations and differing attributed development tasks.

Datasets

• World of Code snapshots V2412, V2510, V2604 — 180 million+ repositories — aggregated from GitHub, GitLab, Bitbucket, and other code forges
• AIDev dataset — 932,791 agent-authored pull requests across 116,211 GitHub repositories — public PR-based census from prior work

Baselines vs proposed

• Bot account-only detection for Claude Code: 28,154 commits vs multi-method union: 850,157 commits (30× gain)
• Dependabot baseline (sanity check): detected 475,379 commits via bot account lookup
• PR-based AIDev vs commit-based detection for Codex: 814,522 PRs vs 843 commits
• PR-based AIDev vs commit-based detection for Claude Code: 5,137 PRs vs 850,157 commits
• OpenHands commits in V2510: 20,863 vs 25,676 in V2604 (23% growth)
• CodeRabbit commits grew 117× from 297 in V2510 to 34,940 in V2604

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.24429.

Fig 1: Overview of the multi-method AI-agent detection framework.

Fig 2: Monthly AI-attributed commit volume by agent across V2412–V2604. Total agent activity rises

Fig 3 (page 3).

Fig 4 (page 3).

Fig 5 (page 3).

Fig 6 (page 3).

Fig 7 (page 3).

Limitations

• Absolute recall of the detection methods is unknown; the union provides only a lower bound causing prevalence underestimation.
• Detection depends on agent signature conventions that evolve rapidly, requiring continual re-calibration and risking missed detections.
• No formal adversarial evaluation to test resilience against agents disguising their traces or obfuscating metadata.
• PR-based and commit-based channels capture different agent populations; thus this work does not present a fully unified measure of all AI agent activity.
• Analysis primarily relies on metadata signals without source-code content analysis; complementary AI-generated code detection methods were not integrated.
• The V2604 snapshot incomplete for commit metadata ingestion, potentially biasing more recent activity estimates.

Open questions / follow-ons

• How can detection methods be adapted or hardened against adversarial evasion where agents disguise or clean commit metadata signatures?
• What is the absolute recall of the combined multi-method approach, and can it be estimated or improved with new signals?
• How do AI agent contributions affect software quality, security vulnerability rates, and project sustainability over time?
• Can complementary content-level AI-generated code detection methods be efficiently integrated with metadata-based detection in large-scale ecosystems?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this paper offers an in-depth example of accurately attributing AI-generated contributions within a complex software ecosystem using a multi-signal, validated detection approach. It demonstrates that relying on a single metadata signal—such as bot account names or commit messages—can lead to severe undercounting and mischaracterization of automated activity. The findings underscore the importance of integrating diverse behavioral and artifact-based signals and carefully validating them, a principle applicable to detecting automated attackers or bots in web systems.

Moreover, the cross-modal disjointness of detection channels (commits vs pull requests) highlights that attackers or agents may present themselves differently depending on operational context, suggesting that defenses need multiple orthogonal detection strategies to fully observe botnet behaviors. The study’s rigorous statistical validation and temporal analyses also illustrate best practices for measuring and tracking automated activity trends over time rather than relying on ad hoc heuristics.

Cite

@article{arxiv2606_24429,
  title={ Detecting AI Coding Agents in Open Source: A Validated Multi-Method Census of 180 Million Repositories },
  author={ Arsham Khosravani and Audris Mockus },
  journal={arXiv preprint arXiv:2606.24429},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.24429}
}

Read the full paper

• arXiv:2606.24429 (abstract)
• arXiv:2606.24429 (PDF)