Motif: Discovering and Automating Personal Web Workflows

Source: arXiv:2607.10531 · Published 2026-07-12 · By Shaokang Jiang, Daye Nam

TL;DR

Motif addresses a fundamental barrier in end-user programming — the "what-to-automate" problem, wherein users often fail to recognize which of their repetitive web interactions can be automated. Prior work assumes users must explicitly identify tasks worth automating and initiate recording or coding, but everyday workflows become invisible and users lack mental models of automation possibilities. Motif introduces ambient automation discovery by passively recording browser activity, mining recurring interaction patterns using LLMs, and generating personalized, deterministic automation programs. Users can review, refine via natural language, and then deploy these automations locally, avoiding ongoing costs and privacy exposure from cloud-based agents.

The study involving eight participants over multiple days showed Motif discovered substantially more automatable patterns than users recognized themselves (average 22 patterns per user, 85% matching actual routines). Users successfully deployed 60% of reviewed candidate programs. Importantly, only 7.5% of those patterns were self-identified by users prior to Motif’s suggestions, confirming the initiation gap. Follow-up surveys indicated strong willingness to continue using Motif-generated programs. The system thus effectively bridges the cognitive and specification gaps limiting end-user automation adoption by surfacing and concretizing latent automation opportunities grounded in real behavior.

Key findings

• Motif discovered on average 22 recurring automatable patterns per participant over 5.5 days of normal browsing, totaling 175 across 8 users.
• Participants recognized only 3 (7.5%) of the 40 patterns reviewed as automatable without Motif’s suggestions, confirming a significant initiation gap.
• 85% (34 out of 40) of the reviewed patterns corresponded to participants’ real daily routines and were deemed useful.
• 60% (24 out of 40) of the attempted Motif-generated program designs were successfully deployed and executed as expected.
• Users rated Motif-generated programs more useful and had higher confidence in them compared to vibe coding attempts.
• Motif’s LLM-based pattern mining outperformed traditional frequency-based sequence mining by identifying semantically meaningful, programmable patterns.
• User refinement via natural language editing of program descriptions enabled improved personalization and generalization of automated workflows.
• All generated automations run locally in the browser extension via Tampermonkey scripts, reducing privacy risk and execution cost compared to cloud agents.

Threat model

The adversary is an untrusted party potentially interested in sensitive user behavior or personal data captured during monitoring. Motif assumes users retain full control over data collection (pause, blacklist URLs) and all data is stored locally. The system does not assume adversaries with active access to LLM APIs or program generation pipelines. The threat model excludes active tampering or adversarial inputs but partially addresses privacy leakage risks through data minimization and filtering.

Methodology — deep read

1. Threat Model & Assumptions: The adversary model centers on privacy and control risks. Motif’s user-controlled browser extension only passively records browser UI, network metadata, OS-level interaction, and screenshots, but excludes sensitive data by filtering passwords and hiding fields, truncating network bodies, and storing data locally. Users have controls to pause monitoring or blacklist URLs. No adversarial or active attacks were modeled.

2. Data: Eight participants used a Chrome extension for 3-7 days (mean 5.5 days) during everyday browsing, collecting extensive multimodal interaction logs, including UI events (clicks, copy-paste, text metadata), tab switches, URLs, screenshots, and lightweight network data within 1.5s after UI events. Data was chunked by inactivity and context windows (~3k tokens) for LLM processing.

3. Architecture / Algorithm: Motif’s pipeline consists of action abstraction, pattern mining, program design, program ranking, and user review.

• Action abstraction: Multimodal LLM (Gemini 3 Flash Preview) summarizes chunks of low-level interactions into higher-level natural-language actions with metadata to avoid hallucination.
• Pattern mining: The LLM identifies recurring programmable sequences from action databases by merging similar patterns, prioritizing meaningfulness over simple frequency.
• Program design: For frequent patterns (occurrence ≥5), LLM generates user-friendly program specifications including trigger mechanisms, descriptive summaries, and notifications.
• Program ranking: Patterns are ranked by frequency and predicted future occurrence likelihood determined by the LLM.
• User review: Users inspect and optionally edit program descriptions, triggers, and notifications via a Chrome extension UI before confirming deployment.
• Program generation: Confirmed designs are sent to the LLM to generate Tampermonkey-compatible JavaScript scripts, explicitly scoping privileges, running locally.

4. Training & Hyperparameters: Specific LLM training was not reported; the system uses a commercially available LLM (Gemini 3 Flash Preview) with low temperature (0.1 or 0.2) for stable deterministic outputs. Pattern mining is triggered every 500 new actions or every 5 minutes for real-time streaming.

5. Evaluation Protocol: The evaluation involved a multi-day field study, followed by a one-hour lab session with each participant:

• Part 1: Passive data collection capturing real browsing.
• Part 2: Lab session including participant reflection on automatable tasks, 10-minute vibe coding attempts, Motif program walkthrough and review of suggested programs, with think-aloud protocol.
• Part 3: Optional longitudinal follow-up survey on continued use. Metrics include number and type of patterns discovered, participant recognition of automatable tasks, program deployment success, user surveys on utility, confidence, and ease of use.

6. Reproducibility: Motif is implemented as a Chrome extension with a native host in Rust; full source code release or dataset availability is not stated. The system depends on proprietary LLM APIs. Study scripts and user materials are not publicly documented.

Concrete example: Alex, a graduate assistant, typically manually refines email replies by switching tabs between Gmail and Google Gemini. Motif observes this repeated sequence, abstracts it, identifies it as a candidate program, generates a program design with a one-click “Refine with Gemini” button inside Gmail, and after user review and slight description edits, generates an executable script replacing the manual multi-step routine.

Technical innovations

• Introduces ambient automation discovery that passively mines users' recurring web interactions without requiring user initiation or explicit task recognition.
• Leverages LLMs not just for code generation but for semantic pattern mining and program design from multimodal user interaction data, surpassing traditional frequency-based sequence mining.
• Combines end-user interaction grounding with natural language program design allowing users to iteratively refine and personalize automations before deployment.
• Generates privacy-preserving, deterministic, cost-efficient automation scripts that run locally within browser extension sandbox rather than relying on expensive LLM agent execution.

Datasets

• User browsing interaction logs — approx 2,000-2,600 actions per participant over 3-7 days — private data collected via Motif extension

Baselines vs proposed

• Vibe coding (user-initiated programming attempts): recognition of automatable patterns = 7.5% (3/40 patterns) vs Motif ambient discovery = 100% (40/40 reviewed patterns discovered)
• Program deployment success rate: vibe coding (not quantitatively reported) vs Motif-generated programs = 60% (24/40 patterns)
• User survey ratings on usefulness: Motif programs scored higher than vibe coding attempts across ease of use, output accuracy, confidence, and usefulness (exact Likert scores not specified)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.10531.

Fig 1: Overview of Motif. Motif supports ambient automation discovery by streaming a pipeline of passive data collection,

Fig 2: User interface. Right top (𝛼−𝛽): controls for managing system monitoring and data collection. Right bottom (1-4):

Fig 3 (page 4).

Fig 4 (page 4).

Fig 5 (page 4).

Fig 6 (page 4).

Limitations

• Small sample size of eight participants limiting statistical generalizability.
• Short data collection duration (3-7 days) may miss infrequent but important workflows like monthly or quarterly reports.
• Limited complexity of automations due to browser extension sandboxing — cannot automate complex UI interactions or system-level tasks.
• Privacy safeguards are minimal and rely on user vigilance; sensitive info filtering is imperfect.
• Dependence on proprietary LLM APIs (Gemini 3) limits full reproducibility and deployment autonomy.
• The vibe coding baseline was limited to a 10-minute coding window, potentially underrepresenting possible user automation capabilities.

Open questions / follow-ons

• How to scale ambient automation discovery to large populations and diverse user contexts while preserving privacy?
• Can local LLMs or federated learning approaches replace proprietary LLM APIs for pattern mining and program generation?
• How to handle long-tail, infrequent automation opportunities that require longer data collection or explicit user input?
• What are effective methods to verify correctness and security of automatically generated automations before deployment?

Why it matters for bot defense

From a bot-defense and CAPTCHA perspective, Motif’s approach highlights how advanced LLM-based pattern mining can reveal complex, personalized user workflows that may otherwise be invisible. Understanding such ambient user behaviors may inform bot-detection heuristics by distinguishing regular repetitive human workflows from scripted bot activity. Moreover, the system’s emphasis on local deterministic programs instead of continuous LLM calls shows a way to reduce server-side inference loads and latency—an important consideration for scalable bot mitigation solutions.

However, Motif also surfaces challenges around privacy and control when passively monitoring detailed user interactions. This underscores the need for CAPTCHA and bot-defense tools to balance deep behavioral analysis with user transparency and opt-in controls. Bot-defense engineers might consider integrating pattern discovery techniques with privacy-preserving mechanisms to detect automation while respecting user autonomy. Lastly, Motif’s ambient discovery paradigm could inspire new UX flows for security prompts or CAPTCHA triggers that adapt by learning from recurring user patterns rather than relying solely on explicit challenge-response interactions.

Cite

@article{arxiv2607_10531,
  title={ Motif: Discovering and Automating Personal Web Workflows },
  author={ Shaokang Jiang and Daye Nam },
  journal={arXiv preprint arXiv:2607.10531},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.10531}
}

Read the full paper

• arXiv:2607.10531 (abstract)
• arXiv:2607.10531 (PDF)