Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining

Source: arXiv:2606.20363 · Published 2026-06-18 · By Yuexing Hao, Xiaomin Li

TL;DR

This paper addresses the challenge of automatically generating explicit skill libraries (SKILL.md) for computer-using agents (CUAs) from interaction trajectory data rather than relying on hand-crafted, manual skill definitions. The authors propose a three-stage pipeline: (1) segmenting GUI trajectories using a simple change-point heuristic, (2) clustering segments into candidate reusable skills with a Wasserstein-distance based metric and refining embeddings via supervised contrastive learning on pseudo-labels, and (3) training a skill-aware policy (Qwen3-8B) via Group Relative Policy Optimization (GRPO) to compose these mined skills for new task completion. The mined skill clusters are shown to be human-readable and align well with ground-truth labels on the source InteraSkill Workflows (IW) benchmark, with five of eight clusters achieving purity ≥ 0.95. However, downstream learned policies only improve skill-step accuracy slightly on IW (18.5% to 20.5%) and do not improve or degrade on transfer benchmarks (BrowseComp+, WebArena). The method underperforms simple frequency-based baselines on key metrics. The study is positioned as a diagnostic analysis revealing that trajectory mining can reveal interpretable skill structure, but the current boundary detection, orderless segment representation, and offline reward model fall short of enabling robust, transferable skill-composition policies.

Key findings

• Using a simple Euclidean-distance threshold on 15-dimensional action features for boundary detection yields 0.419 precision, 0.803 recall, and 0.538 F1 on IW data, but only 0.119 F1 (with IW thresholds) on WebArena transfer data.
• Agglomerative clustering with Bures (Wasserstein) distance on segment mean/variance embeddings achieves normalized mutual information (NMI) of 0.650 and purity ~0.63 at k=8 clusters on IW; supervised-contrastive refinement boosts NMI to 0.862 and purity to 0.837.
• Five of eight skill clusters have purity ≥ 0.95 corresponding to recognizable skill types like document_edit, data_transfer, organize_files on IW data.
• GRPO training of Qwen3-8B raises IW skill-step accuracy only modestly from 18.5% to 20.5%, while BrowseComp+ skill-step accuracy drops slightly from 43.5% to 43.3%; WebArena skill-step accuracy drops from 55.8% to 44.2%.
• Simple baselines outperform learned policies: a Frequency prior baseline scores 34.9% skill-step accuracy on IW, outperforming both MLP and GRPO policies trained on mined skills.
• Auto-generated SKILL.md files outperform hand-written skill tables on some IW data subsets but are consistently worse than Frequency baseline on normalized edit distance across all data sizes.
• Bag-of-actions segment representation (mean and variance of action features) loses ordering information, likely limiting skill composability and downstream policy performance.
• Policy-level transfer to held-out benchmarks (BrowseComp+, WebArena) is weak or negative, showing limited cross-domain generalization of the mined skill vocabulary and offline reward model.

Methodology — deep read

The paper focuses on mining reusable skill libraries for computer-using agents acting on GUI tasks by analyzing recorded trajectories of primitive UI actions (click, type, scroll, copy, paste). The core question is whether mined skills from trajectory clusters can improve downstream policy skill composition and transfer.

1. Threat Model & Assumptions: The adversary in this context is not explicitly modeled since the work targets constructing better agent skill libraries from data. Assumptions include access to labeled trajectory data from a source domain (IW), with ground-truth skill annotations for evaluation, and unlabeled sequences from target domains (BrowseComp+, WebArena) for transfer tests.

2. Data: Source data is the InteraSkill Workflows (IW) dataset, with 2,000 enterprise-style synthetic GUI trajectories containing 8,290 skill-labeled segments. Target sets include WebArena and BrowseComp+ browsing traces without explicit labels. Additional diagnostics include Mind2Web (zero-shot evaluation) and WorkArena-NLP (text-only planning).

3. Architecture / Algorithm: The three-phase pipeline is:

• Phase 1: Trajectory segmentation uses a change-point detection heuristic based on Euclidean distances between successive 15-dimensional primitive action vectors (which encode one-hot action type, screen coordinates, timestamps, text length, and scroll amounts). Threshold θ is selected on IW data to maximize boundary F1. This segments trajectories at large action shifts.
• Phase 2: Skill embedding and clustering summarizes each variable-length segment as the mean and diagonal variance of its constituent action vectors, yielding a 30D vector. Pairwise distances between segments are computed using squared 2-Wasserstein (Bures) distance between diagonal Gaussian approximations. Agglomerative clustering (average linkage) groups segments into candidate skills (k=8 clusters found optimal). A multi-layer perceptron encoder fθ further refines an embedding via supervised contrastive loss trained on the cluster assignments as pseudo-labels to improve clustering purity.
• Phase 3: Skill-aware policy training uses the mined clusters as skill vocabulary labels. A policy based on Qwen3-8B (8B parameter LLM) is fine-tuned with Group Relative Policy Optimization (GRPO), an offline RL method, to generate sequences of skill labels and next primitive actions. The reward model scores skill-labeled responses based on the IW-derived labels.

4. Training Regime: The contrastive encoder trains for 200 epochs, AdamW optimizer with lr=1e-3, batch size 256 on CPU. GRPO fine-tuning uses a single epoch over 1,275 prompts with 4 NVIDIA H200 GPUs and very low learning rate (5e-6).

5. Evaluation Protocol: Segmentation evaluated by precision, recall, F1 of predicted boundaries vs ground truth. Clustering evaluated by normalized mutual information (NMI), purity, silhouette score. Skill composition evaluated by per-step skill prediction accuracy, exact skill sequence match, and normalized edit distance. Transfer evaluated as zero-shot skill-step accuracy on WebArena and BrowseComp+. Baselines include zero-shot base models, a Frequency prior, MLP and Transformer skill-sequence predictors.

6. Reproducibility: The code is available in an anonymous Github repository. The IW dataset is synthetic and not publicly detailed, WebArena and BrowseComp+ are public benchmarks. Model weights for Qwen3-8B are presumably closed.

Example end-to-end: For a trajectory of clicks, types, and scrolls on IW, the heuristic detects boundaries at large action jumps. Each segment's mean/variance action vector is computed, distances between segments calculated by Wasserstein metric, clusters formed by agglomerative clustering. Pseudo-labels train an MLP embedding with supervised contrastive loss. Then Qwen3-8B is fine-tuned via GRPO to predict skill sequences, evaluated by skill-step accuracy and compared with baselines. The learned skills are human-interpretable but fail to improve next-step prediction substantially beyond frequency priors or simple MLP/Transformer models.

Technical innovations

• Applying squared 2-Wasserstein (Bures) distance between diagonal Gaussian segment embeddings for clustering GUI action segments into skills.
• A three-phase pipeline combining simple change-point boundary detection, Wasserstein clustering, and supervised-contrastive embedding refinement for automated skill library generation from GUI trajectories.
• Using Group Relative Policy Optimization (GRPO) offline reinforcement learning with a learned trajectory reward model to train a skill-aware policy from mined skill annotations.
• Demonstrating that readable skill clusters with high purity can be induced without manual human labeling, but that segment embeddings based on mean/variance destroy sequential ordering crucial for downstream transfer.
• Empirically showing with extensive baselines that mined skill vocabularies do not improve cross-domain skill sequencing compared to trivial frequency priors or MLP/Transformer models.

Datasets

• InteraSkill Workflows (IW) — 2,000 synthetic enterprise-style GUI trajectories with skill labels — source domain
• WebArena — browsing traces — public benchmark, held-out transfer domain
• BrowseComp+ — browsing traces — public benchmark, held-out transfer domain
• Mind2Web — web-action context data for zero-shot diagnostics — public benchmark
• WorkArena-NLP — text-only structured planning examples derived from WorkArena — diagnostic

Baselines vs proposed

• Frequency baseline on IW skill-step accuracy — 34.9% vs GRPO Qwen3-8B — 20.5%
• MLP skill-sequence predictor on IW — 23.3% skill-step accuracy vs GRPO Qwen3-8B — 20.5%
• Transformer skill-sequence predictor on IW — 34.6% skill-step accuracy vs GRPO Qwen3-8B — 20.5%
• GRPO Qwen3-8B IW skill-step accuracy — 20.5% vs zero-shot Qwen3-8B — 18.5%
• GRPO Qwen3-8B BrowseComp+ skill-step accuracy — 43.3% vs zero-shot Qwen3-8B — 43.5%
• GRPO Qwen3-8B WebArena skill-step accuracy — 44.2% vs zero-shot Qwen3-8B — 55.8%
• Auto-generated SKILL.md normalized edit distance on IW subsets better than hand-written in some cases but worse than Frequency baseline at all sizes (Frequency = 0.485 vs Auto-SKILL.md = 0.528 at N=2000)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.20363.

Fig 1: Study design for automated SKILL.md generation. IW is the source dataset for trajectory

Fig 2: Data-efficiency comparison for generated SKILL.md. Lower normalized edit distance

Fig 3 (page 16).

Fig 4 (page 18).

Fig 5 (page 18).

Fig 6 (page 19).

Fig 7 (page 20).

Fig 8 (page 21).

Limitations

• The simple Euclidean-distance boundary detector over-segments skills due to within-skill action variation; it is not domain-stable and does not transfer zero-shot effectively.
• Segment representation ignores sequential order of actions by summarizing segments as mean and variance, losing execution-relevant information for skill composition.
• The clustering and embedding refinement rely on pseudo-labels from initial clusters, not fully unsupervised, potentially overfitting to source biases.
• The learned skill-aware policy trained with group relative policy optimization (GRPO) shows minimal improvement and underperforms trivial statistical baselines.
• Transfer to held-out benchmarks (BrowseComp+, WebArena) fails or degrades performance, indicating weak cross-domain generalization of mined skills and reward model.
• The IW dataset is synthetic and may lack real-world complexity; transfer evaluation lacks live environment interaction or online RL feedback.
• The policy training uses offline reward models without supervised warm start or direct supervised sequence loss baselines for comparison, obscuring source of failures.

Open questions / follow-ons

• Can learned boundary detectors leveraging action-prediction errors improve skill segmentation precision and transfer over simple heuristics?
• Would order-aware segment representations that preserve action sequences enable better downstream skill composition and transfer?
• How does joint training of skill segmentation, clustering, and reward modeling affect transferability and policy improvement?
• Can combining mined skill libraries with supervised skill definitions or transition priors yield more robust and generalizable policies?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, the paper demonstrates that while mining reusable action primitives or 'skills' from user interaction trajectories can yield interpretable and human-inspectable behavioral clusters, this clustering alone does not guarantee improved downstream policy performance or robust cross-domain transfer. This highlights that detected skill units must carefully capture temporal order and contextual nuances to enable reliable compositional behavior, a critical factor when modeling or detecting sophisticated automated agents attempting to mimic human workflows.

Furthermore, the finding that simple statistical frequency baselines outperform complex learned skill-composition models suggests that in bot detection or adaptive CAPTCHA generation, leveraging straightforward pattern frequency and transition statistics might be more effective and robust than relying solely on learned abstract skill representations. The diagnostic nature of this study also encourages careful validation of any automated user skill decomposition approach against naive baselines and in diverse environments before deployment in security-sensitive contexts.

Cite

@article{arxiv2606_20363,
  title={ Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining },
  author={ Yuexing Hao and Xiaomin Li },
  journal={arXiv preprint arXiv:2606.20363},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.20363}
}

Read the full paper

• arXiv:2606.20363 (abstract)
• arXiv:2606.20363 (PDF)