VISUALSKILL: Multimodal Skills for Computer-Use Agents

Source: arXiv:2606.18448 · Published 2026-06-16 · By Ziyan Jiang, Li An, Yujian Liu, Jiabao Ji, Qiucheng Wu, Jacob Andreas et al.

TL;DR

This paper addresses the limitations of current computer-use agents (CUAs) that interact with graphical user interfaces (GUIs). Although CUAs can approach human-level performance on benchmarks, they struggle with long-horizon tasks and generalization to unseen software largely due to limited application-specific procedural knowledge. Prior work encodes such knowledge as reusable text-only skills, but the authors argue that this neglects the inherently visual nature of GUIs, where screenshots and spatial layouts provide critical information. To tackle this, they propose VISUALSKILL, a multimodal, hierarchical skill representation that retains visual figures alongside textual procedures, organized as a per-application central index with per-topic files. At inference time, an agent consults these on-demand via a load_topic tool that delivers relevant text and images together.

VISUALSKILL skills are constructed via a two-stage pipeline: first mining authored documentation with vendor-supplied screenshots, then augmenting those with screenshots and UI notes automatically captured by live UI exploration controlled by an LLM agent. Evaluations on two CUA benchmarks (CUA-World and OSExpert-Eval) demonstrate that VISUALSKILL outperforms a no-skill baseline by +15.3% absolute average score and a matched text-only skill (derived from the same source content) by +8.3% absolute, pinpointing that visual grounding materially improves UI element identification and state verification in multi-step workflows. Detailed ablations show both stages of skill construction and the on-demand skill loading mechanism are critical to the performance gains.

Key findings

• VISUALSKILL lifts average task success score from 0.303 (no skill) to 0.456 (+15.3% absolute) across 177 tasks on CUA-World and OSExpert-Eval.
• VISUALSKILL outperforms matched text-only skill by +8.3% absolute (0.456 vs 0.373) at Stage 2 UI exploration.
• Stage 1 skill (mined from documentation) alone improves from 0.303 to 0.363 (+6.0%), Stage 2 UI exploration adds +9.3% absolute on top.
• Multimodal advantage concentrates on identifying UI elements and verifying intermediate states after each action, where text descriptions are ambiguous.
• MCP tool-based on-demand loading delivers ∼10× more figures per task vs direct Read, improving task score by 3–4% absolute.
• Combining free UI exploration and targeted trajectory-based exploration yields super-additive skill improvement (0.504 score combined vs 0.462 or 0.453 for subpasses).
• VISUALSKILL yields largest gain (up to +16.6% absolute) on visually intensive domains like GIMP and OpenToonz.
• In short tasks with explicit actions, figures confer little advantage, narrowing modality gaps.

Threat model

N/A — This work does not focus on adversarial threat modeling but rather on improving the representation and retrieval of application-specific procedural knowledge for computer-use agents interacting with graphical user interfaces.

Methodology — deep read

Threat Model & Assumptions: The work assumes an environment where a computer-use agent (CUA) interacts with graphical user interfaces by observing screenshots and issuing low-level mouse and keyboard actions. The adversary here is not explicitly modeled as it focuses on agent knowledge representation and retrieval. Assumed is that application-specific procedural knowledge is not embedded in pretraining and must be externally supplied.

Data: Evaluation is conducted on two benchmark suites—CUA-World (covering five application domains, including LibreOffice Writer, Calc, Impress, QGIS, OpenToonz) and OSExpert-Eval (three domains: LibreOffice, GIMP, Tableau). Task sets have per-application train/test splits. Source content for skills comes from authored documentation (PDF or HTML manuals) and live UI exploration screenshots. Total evaluation covers 177 tasks. Verifiers provided by each benchmark score task success.

Architecture / Algorithm: VISUALSKILL is a hierarchical multimodal skill artifact per target application. It consists of a centralized skill.md index listing per-topic skill guides. Each guide contains a text body plus associated UI figures/screenshots. The agent consults the skill via a load_topic MCP tool, which fetches the text and images for a requested topic on demand during execution, avoiding upfront loading of the full skill. A matched text-only control skill is built jointly from the same source but replaces figures with textual descriptions.

Training / Construction Pipeline: The two-stage pipeline constructs VISUALSKILL skills. Stage 1 mines the official application documentation, parsing the table of contents into topics. For each topic, text bodies and vendor-provided figures are extracted. Stage 2 enriches the skill by running an LLM-based UI explorer agent to partition the live UI into regions (free exploration) and target UI regions that caused failures in training trajectories (trajectory-targeted exploration). Workers capture cropped screenshots and notes per region. An assembler agent integrates these UI captures into the corresponding topic guides, updating text and appending figures. Both Smm (multimodal) and Stxt (text only) variants are produced in parallel, ensuring shared procedural content to isolate effects of multimodality.

Training Regime: This is primarily a data construction and evaluation framework, not supervised training. The agent used for evaluation is Claude Code CLI backed by Claude Opus 4.6. Benchmarks provide per-task budget limits (40-200 max steps or 15 min per-task wall-clock cap). The LLM-based planners and workers run in Docker containers. Joint-generation calls produce both multimodal and text-only topic guides from the same inputs.

Evaluation Protocol: Metrics are benchmark-specific normalized success scores in [0,1]. All agent aspects except skill artifact are held fixed. Five conditions compared: no skill, Stage 1 text-only skill, Stage 1 VISUALSKILL, Stage 2 text-only skill, Stage 2 VISUALSKILL. The critical comparison is VISUALSKILL vs matched text-only within each stage. Ablations probe the effect of the MCP loading tool vs direct Read calls, and incremental contributions of Stage 1 and sub-passes within Stage 2 exploration. Qualitative analysis identifies failure modes addressed by visual figures.

Reproducibility: Code and skill construction pipeline are open-sourced at https://github.com/XMHZZ2018/VisualSkills. Benchmarks and datasets are public as per citations. Full details and prompt excerpts are in appendices. Frozen weights for Claude Opus 4.6 are not released by authors but accessible via Anthropic.

End-to-End Example (Stage 1 topic): The documentation’s table of contents defines a skill topic "formatting-text/character-formatting." The corresponding pages are scanned. Figures and text are jointly processed by an LLM call that generates the multimodal guide with embedded UI screenshots and a text-only guide that describes the visual elements verbally. At inference, the agent encounters a task step requiring use of this topic, calls load_topic("formatting-text/character-formatting"), and receives both text and figure blocks inline. The agent uses the figure to directly localize the target icon to click, verifying visual state after the action.

Overall, the methodology tightly integrates multimodal skill construction from heterogeneous sources, hierarchical skill organization, an on-demand retrieval mechanism, and rigorous paired evaluation to isolate the benefit of retaining visual figures in computer-use agent skills.

Technical innovations

• Hierarchical multimodal skill representation with per-application centralized index and per-topic text-plus-figure guides for CUAs.
• Two-stage pipeline combining authored documentation mining and LLM-controlled live UI exploration to enrich skill artifacts with up-to-date visual knowledge.
• On-demand MCP tool load_topic interface for skill loading, delivering both relevant text and visual figures inline during agent rollout.
• Joint generation of matched multimodal and text-only skill variants from identical source content to isolate modality effects.

Datasets

• CUA-World — 167 tasks across 5 domains (LibreOffice Writer, Calc, Impress, QGIS, OpenToonz) — public benchmark
• OSExpert-Eval — 110 tasks across 3 domains (LibreOffice, GIMP, Tableau) — public benchmark
• Vendor-authored documentation (PDF/HTML manuals) per application — public or shipped with software
• Captured UI screenshots from live application exploration — generated in this work

Baselines vs proposed

• No skill baseline: mean score = 0.303 vs VISUALSKILL Stage 2 multimodal: 0.456 (+15.3% absolute)
• Stage 2 VISUALSKILL text-only control: 0.373 vs VISUALSKILL multimodal: 0.456 (+8.3% absolute)
• Stage 1 VISUALSKILL multimodal: 0.363 vs Stage 1 text-only: 0.344 (+1.9% absolute)
• MCP load_topic tool delivery: Writer score 0.276 vs Direct Read 0.236 (+4.0% absolute)
• Stage 2(a) free UI exploration VISUALSKILL: 0.462 vs Stage 1 baseline 0.436 (+2.6% absolute)
• Stage 2(b) trajectory-targeted exploration VISUALSKILL: 0.453 vs Stage 1 baseline 0.436 (+1.7% absolute)
• Combined Stage 2 VISUALSKILL: 0.504 vs text-only control 0.451 (+5.3% absolute)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.18448.

Fig 1: A text-only skill struggles to describe the

Fig 2: The two-stage VISUALSKILL construction pipeline. Stage 1 parses the authored documentation into a

Fig 5: VISUALSKILL and its text-only control on topic t = formatting-text/character-formatting. Both

Fig 6: The literal tool-result structure for one topic, in both variants. Left (Smm): the content list alternates

Fig 9: Two of the worker’s twelve captures for region

Fig 10: Initial state of exam_paper_formatting.

Fig 11: Worker capture, step 4: the ▼arrow has been

Fig 12: Worker capture, step 38: after selecting

Limitations

• Current models do not reliably select which figures within loaded topics are relevant, leading to coarse on-demand loading with inflated context cost.
• Weaker base models suffer from context length pressure when using multimodal skills, sometimes performing worse than no skill (e.g., QWEN3.5-397B on OSWorld LibreOffice).
• The two-stage pipeline depends on availability and quality of authored documentation; domains with sparse or outdated manuals may have limited Stage 1 coverage.
• Evaluation lacks adversarial robustness tests or out-of-distribution software beyond provided benchmarks.
• The study uses only Claude Opus 4.6 agent; results may vary with other LLM architectures or versions.
• Skill loading granularity is fixed at per-topic level; more fine-grained visual content retrieval strategies remain unexplored.

Open questions / follow-ons

• How to develop selective figure retrieval or attention mechanisms to reduce context length burdens while maintaining multimodal benefits?
• Can the VISUALSKILL approach be generalized to dynamic web applications and mobile app UIs where visual state changes rapidly?
• What are the impacts of adversarial or corrupted visual inputs on multimodal skill performance, and how to harden agents?
• Is it possible to automatically generate or update skills continuously from user interaction logs or video walkthroughs rather than only documentation and exploration?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners building computer-use agents or automated UI-driven bots, VISUALSKILL demonstrates the substantial benefit of integrating visual artifacts alongside procedural text to improve reliability in complex GUI environments. Retaining figures helps disambiguate target UI elements and verify intermediate states, crucial for long-horizon multi-step workflows.

This suggests that defenses or detection strategies relying on agents’ misunderstandings of visual UI context may face greater challenges as agents incorporate multimodal skills. The paper also highlights the importance of specialized skill retrieval mechanisms (like MCP tools) that allow efficient access to multimodal knowledge during deployment. Overall, VISUALSKILL provides insights into how richer, structured multimodal knowledge bases can empower CUAs, and may inform design of CAPTCHAs or bot tests that exploit visual grounding nuances.

Cite

@article{arxiv2606_18448,
  title={ VISUALSKILL: Multimodal Skills for Computer-Use Agents },
  author={ Ziyan Jiang and Li An and Yujian Liu and Jiabao Ji and Qiucheng Wu and Jacob Andreas and Yang Zhang and Shiyu Chang },
  journal={arXiv preprint arXiv:2606.18448},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.18448}
}

Read the full paper

• arXiv:2606.18448 (abstract)
• arXiv:2606.18448 (PDF)