RepoTrace: Browser-Assisted Evidence Collection for GitHub Research Datasets

Source: arXiv:2607.05106 · Published 2026-07-06 · By Xue Yao, Zehua Zhang, Jiatong Liu, Yongqiang Tian

TL;DR

RepoTrace addresses a key challenge in empirical software engineering research, where datasets are frequently built manually from GitHub issues and pull requests but the evidence, coding decisions, and rationale become fragmented across browsers, spreadsheets, and notes. This fragmentation undermines data provenance, update tracking, and auditing multi-review labeling decisions. RepoTrace introduces a browser-assisted local workspace that unifies page snapshots, comments, labels, notes, screening and labeling decisions, refresh history, and exports for GitHub issues and pull requests. This integration preserves the source evidence alongside research interpretations in an auditable, queryable form.

The tool combines a Chrome side panel extension for evidence extraction, an Express backend with SQLite for local persistence, and a React dashboard for review, labeling, and exporting datasets. Validation on 20 matplotlib issues showed RepoTrace preserved 22 snapshots, 38 comments, 20 notes, 98 annotations, 20 screening reviews, and handled simulated multi-reviewer conflicts. RepoTrace supports comprehensive local workflows from collection to consensus to refresh to export, demonstrating feasibility for manual and semi-manual GitHub research dataset construction.

Key findings

• RepoTrace preserved 22 snapshots, 38 comments, 20 research notes, 98 annotations, 20 screening reviews, and 20 fix-evidence entries from 20 GitHub issues across two projects (backend/rendering and regression/compatibility).
• The tool successfully injected and surfaced 4 synthetic multi-reviewer consensus conflicts in the regression/compatibility project, demonstrating conflict detection and review workflow.
• RepoTrace's refresh mechanism detected changes in GitHub issues, preserved new snapshots, logged change histories, and flagged records for re-review before export.
• The automated test suite comprised 37 tests covering URL detection, data extraction, backend APIs, duplicate handling, exports, backups, consensus workflows, audit logs, and dashboard rendering.
• RepoTrace’s scoped export produces single-record JSON packages, filtered record exports, or full project backups that include evidence, labels, annotations, and provenance.
• The SQLite schema enforces project-scoped uniqueness by repository owner, name, record type, and number, and stores multi-snapshot histories and audit trails.
• The system architecture integrates a Chrome Manifest V3 extension (content script + side panel), Express backend, SQLite database, React dashboard, and shared TypeScript types for consistent contracts.
• RepoTrace runs fully locally, requiring only Node.js, npm, SQLite, and a Chromium browser, supporting reproducibility and offline usage.

Threat model

Empirical software engineering researchers manually curating GitHub datasets face challenges of fragmented evidence, untracked updates, and unanalyzable coding provenance. RepoTrace defends against these issues by integrating browser-extracted evidence, codes, and annotations in a single auditable local workspace. The adversary modeled is not a malicious attacker but data fragmentation and loss of provenance through manual workflows. RepoTrace assumes no adversary capabilities beyond natural changes on GitHub or researcher error; there is no protection against hostile tampering or data poisoning.

Methodology — deep read

RepoTrace's threat model centers on empirical software engineering researchers manually curating GitHub issue and pull request datasets. The adversary is not a hostile attacker but rather the risk of provenance loss, fragmented data, and untracked updates during manual collection. RepoTrace assumes researchers need to link evolving GitHub discussions with their research labels and rationale reliably.

Data used for validation included 20 issues from the matplotlib/matplotlib repository split into two projects: one on backend/rendering issues and another on regression/compatibility issues. Validation included injecting 4 synthetic multi-reviewer consensus conflicts to test conflict workflows. Dataset fields collected included issue title, body, labels, comments, timestamps, snapshots (raw text and HTML), research notes, annotations, screening decisions, fix evidence, update history, and audit logs.

RepoTrace comprises four components: a Chrome extension detecting and extracting visible evidence from GitHub issue/pull-request pages into a side panel UI; an Express backend API persisting data to a local SQLite database with roughly 20 tables covering records, snapshots, annotations, labels, notes, reviews, audit logs, consensus states, refresh history, and change logs; a React dashboard for project management, filtering, labeling, consensus review, quality checks, and export; and a shared TypeScript types package defining common contracts.

Collection is browser-first: The extension side panel auto-extracts repository metadata and page fields visible on GitHub pages, allowing selective inclusion of title, body, labels, comments, snapshots, notes, and research labels. GitHub API enrichment is optional and used only for refresh, without requiring authentication tokens.

Labeling distinguishes GitHub source labels from research labels that encode study-specific coding schemes. The system supports taxonomy evolution with tracking of label additions and backfill tasks for existing records. Multi-reviewer annotations track labeler identity, confidence, rationale, and status, with conflicts surfaced in the dashboard for consensus review.

Refresh and audit track GitHub record changes through manual single-record or project-level checks. Changed records generate new snapshots, log diffs, and flag updates for re-inspection before dataset export.

Exports include single-record JSON packages, filtered sets using dashboard queries, and full project backups preserving all metadata and provenance.

The automated test suite verifies extraction from GitHub URLs, database initialization, backend APIs, duplicate detection, export correctness (including round-trip backups), audit logging, and UI rendering.

One concrete usage example was the collection of an issue from matplotlib: The Chrome extension side panel detects and loads the issue page metadata and comments, extracts them into the local SQLite backend with a timestamped snapshot, then the researcher adds research labels, notes, and screening decisions in the dashboard. If new activity occurs on GitHub, the researcher runs a refresh to append a new snapshot and audit log for the changed record. Finally, the project exports a scoped JSON dataset including evidence, labels, and audit history.

The source code and demo dataset are publicly available with reproducibility instructions, including seeding demo data, running tests, performing update checks, and exporting JSON datasets.

Some implementation details remain unclear from the paper, such as specific indexing strategies or performance measures on very large datasets, and how taxonomy backfill scales, which may be explored in future work.

Technical innovations

• A browser-first local evidence capture workflow that preserves full rendered GitHub issue/pull-request page snapshots alongside research labels and notes.
• Integration of a Chrome side-panel extension, Express backend with SQLite, and React dashboard into a unified local workspace maintaining provenance, audit trails, update history, and consensus workflows.
• Support for evolving research label taxonomies with explicit backfill queues and multi-reviewer conflict detection enabling auditable consensus resolution.
• Scoped dataset exports separating single-record evidence packages, filtered record sets, and full project backups retaining complete provenance and intermediate coding decisions.

Datasets

• Matplotlib Issues Dataset — 20 GitHub issues from matplotlib/matplotlib — self-collected for validation

Limitations

• No user study or controlled evaluation on researcher productivity or error reduction; validation is a functionality demonstration on 20 issues.
• Synthetic multi-reviewer conflicts were injected rather than observed real-world disagreements, limiting insight into workflow under realistic noisy coding.
• No large-scale performance analysis or scalability assessment on bigger or more complex repositories.
• The refresh mechanism is manual, not continuous synchronization; thus may not capture rapidly changing records without user intervention.
• The paper does not evaluate integration with existing mining tools or qualitative coding platforms directly.
• Taxonomy backfill and multi-review consensus workflows may incur workflow overhead not quantitatively assessed.

Open questions / follow-ons

• How does RepoTrace scale when applied to large repositories with thousands of issues and frequent updates?
• Can the taxonomy evolution and backfill workflow be optimized or automated to reduce reviewer overhead in large studies?
• What is the impact of RepoTrace usage on researcher productivity, coding consistency, and error rate in practice?
• How might integration with other mining tools or qualitative coding platforms enhance or complement RepoTrace?

Why it matters for bot defense

While RepoTrace is not a bot defense or CAPTCHA system, its approach to tightly linking source web evidence, annotations, and provenance in a unified local workspace has conceptual parallels to bot defense research that requires audit trails and multi-review coordination. Bot-defense engineers might draw inspiration from RepoTrace’s browser-assisted evidence capture and change tracking to build tools that preserve contextual evidence around suspect bot activity for forensic analysis. The taxonomy evolution and conflict resolution features could also guide multi-annotator workflows for labeling bot traffic versus human interaction. However, the focus on manual qualitative coding limits direct applicability to automated CAPTCHA or bot detection methods. Nonetheless, the principles of provenance preservation, auditability, and incremental evidence refresh could inform standards for labeling and validating bot detection datasets.

Cite

@article{arxiv2607_05106,
  title={ RepoTrace: Browser-Assisted Evidence Collection for GitHub Research Datasets },
  author={ Xue Yao and Zehua Zhang and Jiatong Liu and Yongqiang Tian },
  journal={arXiv preprint arXiv:2607.05106},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.05106}
}

Read the full paper

• arXiv:2607.05106 (abstract)
• arXiv:2607.05106 (PDF)