Social-Annotate: Self-Healing Browser Extension to Annotate and Collect Social Media Data

Source: arXiv:2607.01460 · Published 2026-07-01 · By Ali Najafi, Ismail Uluturk, Onur Varol

TL;DR

Social-Annotate addresses the critical challenge of collecting high-quality human-annotated social media data while preserving ecological validity. Traditional annotation pipelines typically separate annotation tasks from the native social media environment, which can introduce bias and operational overhead. This browser extension injects customizable survey forms directly into social media and messaging platforms' webpages, enabling users to provide annotations in-context as they naturally interact with content. Importantly, because DOM structures on these platforms frequently change, Social-Annotate integrates a self-healing agent powered by large language models (LLMs) that autonomously detects interface changes, generates updated selectors, and validates injection in a live browser environment. This combination reduces developer maintenance and keeps the tool robust over time across 12 supported platforms, including X, Instagram, TikTok, WhatsApp, and Telegram.

Experimentally, the self-healing agent proved capable of extracting valid selectors from archived snapshots of X/Twitter from 2010 to 2026 and Bluesky from 2023 and 2026, achieving complete metadata capture in most cases. For other platforms, the agent successfully generated selectors on live snapshots without prior knowledge. The extension offers a no-code survey design interface for non-technical users and supports exporting collected annotations as JSONL files or via API endpoints for multi-annotator setups. It also supports intervention studies by allowing dynamic manipulation of displayed content using external NLP models or precomputed mappings. Social-Annotate thus significantly lowers the technical barrier and maintenance burden in collecting ecologically valid, richly annotated social media datasets.

Key findings

• Social-Annotate supports injection of customizable annotation forms directly into native platforms across 12 social and messaging networks, including X, Instagram, TikTok, WhatsApp, and Telegram.
• A self-healing agent powered by large language models autonomously detects and adapts to DOM structure changes, maintaining valid selectors without manual intervention.
• The LLM-based self-healing pipeline achieved full annotation functionality on all five evaluated archived X/Twitter snapshots (2010, 2014, 2017, 2020, 2026) with complete metadata capture in 4/5 cases.
• Similarly, selector extraction succeeded on Bluesky snapshots from 2023 and 2026, validating robustness across time.
• On live snapshots, the agent successfully generated selectors for Reddit, Mastodon, WhatsApp, Telegram, and Truth Social without predefined selectors.
• The platform’s no-code survey design interface enables configuration of diverse question types via JSON schemas usable by non-technical researchers.
• Annotation data can be exported as JSONL files or sent to specified API endpoints to support distributed annotation workflows.
• The intervention feature allows content manipulation in Blind or Aware modes through external REST APIs or precomputed mappings, facilitating experimental studies on influence of content variations.

Threat model

The threat model centers on the challenge of maintaining robust annotation form injection despite adversarial or non-adversarial changes in social media platform front-end structures. The adversary is effectively the dynamic platform UI that can break brittle CSS selectors, causing data loss. Researchers cannot rely on official API access due to restricted or privatized endpoints. The system assumes no direct collaboration from platforms and limited prior knowledge of future DOM changes. Adversaries cannot prevent browser extension injection entirely but may indirectly cause failure by modifying HTML structures.

Methodology — deep read

1. Threat model & assumptions: The authors assume a researcher or annotation organizer who wants to collect human-labeled data directly on social media platforms. The adversary is not explicitly studied; rather, the technical challenges addressed are due to frequent DOM/UI changes that break extension functionality. The system must operate under evolving proprietary front-end changes without platform cooperation (no official APIs).

2. Data: To evaluate robustness, archived HTML snapshots were collected from the Internet Archive's Wayback Machine for X/Twitter (years 2010, 2014, 2017, 2020, 2026) and Bluesky (2023, 2026). Live snapshots of current pages from Reddit, Mastodon, WhatsApp, Telegram, and Truth Social were also used. Labels are the extracted DOM selectors and survey injection success metrics. No user-generated annotations or sensitive data are involved in selector extraction evaluation.

3. Architecture/algorithm: The core system is a browser extension built for Chromium-based browsers with multiple components: manifest.json for permissions; background scripts for event handling; content scripts specific to each platform to inject forms; popup UI for control; and options pages for configuration. The novel self-healing agent uses a three-component pipeline: a platform abstraction layer defining typed selector schemas with Pydantic; an LLM client (backends like Claude or Gemini) that extracts selectors from raw HTML using constrained schema outputs; and a live browser environment implemented via Playwright to load the extension with proposed selectors and verify injection through dynamic DOM mutation observers and simulated form submission. If offline validation or injection fails, the LLM retries with updated prompts for self-correction.

4. Training regime: The LLMs are used in zero-shot or few-shot prompting modes guided by schema and platform-specific context. The extraction process involves token-limited HTML pruning and iterative retry until selector sets pass schema validation and browser injection tests. Details on prompts, epoch, or batch training are not applicable as the models are used as APIs.

5. Evaluation protocol: Evaluation focuses on the self-healing agent’s success rate in extracting valid selectors and enabling complete annotation injection and metadata capture. Offline static HTML validation and live dynamic browser tests were conducted. Five archived X/Twitter pages and two Bluesky pages were tested, plus live snapshots for other platforms. Results report success/failure of per-platform selector adaptation and ability to conduct annotation. Limitations related to very large HTML inputs (LinkedIn) and dynamic JavaScript-triggered parts (e.g., Instagram comments) were noted.

6. Reproducibility: The extension is open source on GitHub (github.com/ViralLab/social-annotate), providing the tool, self-healing agent implementation, and recipes to extend platforms. The evaluation relies on publicly available archived snapshots and live pages, though practical reproduction depends on availability of LLM APIs used (e.g., Claude). Pretrained LLM weights or datasets are not released as models are accessed via APIs.

Example end-to-end: Given an archived snapshot of an X/Twitter page from 2017, the self-healing agent loads its HTML, prunes for token limits, and submits it with platform schema to the LLM. The model proposes candidate CSS selectors that are offline validated against the snapshot. If valid, these selectors are loaded into a live Chromium environment with the extension installed. The agent simulates user scroll and interacts with injected surveys to confirm successful injection and data collection. If the selectors fail any step, the LLM is prompted again. Once selectors pass all checks, the configuration is staged for operator approval and deployment. Thus, Social-Annotate achieves autonomous, live-verified selector extraction ensuring robustness despite platform evolution.

Technical innovations

• Integration of a self-healing agent powered by large language models that autonomously detects and repairs broken DOM selectors for survey injection on evolving social media platforms.
• A three-phase selector extraction pipeline combining offline HTML validation, LLM-guided extraction, and live browser environment verification with user interaction simulation.
• A no-code, JSON-schema based survey form design interface allowing non-technical users to configure customizable annotation tasks directly within native social media sites.
• Support for dynamic intervention studies through content manipulation modules enabling on-the-fly alteration of displayed posts and user profiles using external NLP models or precomputed mappings.

Datasets

• X/Twitter archived snapshots — 5 snapshots from years 2010, 2014, 2017, 2020, 2026 — Internet Archive Wayback Machine
• Bluesky archived snapshots — 2 snapshots from 2023 and 2026 — Internet Archive Wayback Machine
• Live page snapshots — Reddit, Mastodon, WhatsApp, Telegram, Truth Social (size unspecified) — collected by authors

Baselines vs proposed

• No explicit baseline tools reported for selector extraction; comparison qualitative against manual maintenance effort.
• Self-Healing Agent on X/Twitter snapshots: full annotation injection success on all 5, with complete metadata capture on 4 (missing timestamp attribute in 2010 snapshot).
• Self-Healing Agent on Bluesky snapshots: full success on both 2023 and 2026 snapshots.
• Self-Healing Agent on live snapshots: successful injection and selector extraction on Reddit, Mastodon, WhatsApp, Telegram, and Truth Social snapshots.

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.01460.

Fig 1: Social-Annotate platform design schematic. Components of the extension presented in the middle panel

Fig 2: Social-Annotate options page. The figure presents different forms or interfaces to setup a study and

Fig 3 (page 11).

Fig 4 (page 13).

Limitations

• LinkedIn platform excluded from evaluation due to LLM input size limitations caused by very large HTML pages.
• Instagram comment section DOM elements dynamically loaded via JavaScript after user interaction, complicating static snapshot testing.
• Self-healing agent currently relies solely on selector extraction; more complex DOM changes requiring logic/code generation remain future work.
• Evaluations based on archived and live page snapshots rather than real-time continuous deployment, unknown performance under rapid mid-session UI changes.
• LLM dependency introduces external API costs, possible latency, and reliance on proprietary models (Claude, Gemini).
• Ethical and privacy considerations require careful informed consent for human-subject studies, though handled via configurable forms.

Open questions / follow-ons

• How can the self-healing approach be extended to handle DOM mutations requiring procedural logic or complex code rather than simple selector identification?
• Can local, client-side AI models be integrated to perform selector regeneration and content manipulation without external API dependencies, enhancing privacy and latency?
• What methods can improve scalability and robustness for very large or highly dynamic platforms (e.g. LinkedIn, Instagram) that exceed token or static snapshot processing limits?
• How can annotation data quality be systematically validated or augmented when collected via dynamic in-browser forms, especially under variable user engagement?

Why it matters for bot defense

This work is highly relevant to bot-defense and CAPTCHA practitioners engaged in labeling and collecting human interaction data on social platforms. By embedding annotation tasks natively within social media environments, Social-Annotate preserves ecological validity—critical for understanding real user behavior and designing effective bot detection models. The automated selector regeneration mitigates a major engineering bottleneck as sites frequently update UI elements that often break measurement tools. Furthermore, its modular intervention capabilities enable experimental manipulation of content presentation, valuable for studying how adversarial actors exploit media or how CAPTCHAs might influence user engagement and bot detection. Practitioners seeking scalable, low-maintenance annotation solutions would benefit from adopting or extending Social-Annotate’s LLM-powered self-healing and no-code survey design features.

Cite

@article{arxiv2607_01460,
  title={ Social-Annotate: Self-Healing Browser Extension to Annotate and Collect Social Media Data },
  author={ Ali Najafi and Ismail Uluturk and Onur Varol },
  journal={arXiv preprint arXiv:2607.01460},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.01460}
}

Read the full paper

• arXiv:2607.01460 (abstract)
• arXiv:2607.01460 (PDF)