VIA: Visual Interface Agent for Robot Control

Source: arXiv:2607.11119 · Published 2026-07-13 · By Hengyuan Hu, Priya Sundaresan, Jensen Gao, Dorsa Sadigh

TL;DR

This paper introduces VIA (Visual Interface Agent), a novel framework that repurposes off-the-shelf foundation model (FM)-powered agents to control robot manipulators via a visual interface, without any robot-specific fine-tuning or privileged state access. Instead of training specialized vision-language-action models on robotics data, VIA treats robot control like a computer-use task: the agent observes a 3D point-cloud-based browser UI through screenshots and issues intuitive high-level commands via human-legible Model Context Protocol (MCP) tools, such as setting gripper waypoints and executing them via a simple controller. This leverages the general visual perception, reasoning, planning, and closed-loop capabilities of large language models like Claude Code and Codex in zero-shot robot manipulation. The system achieves high success rates across diverse tabletop tasks – 96.7% on LIBERO-Goal tasks and 100% on a long-horizon block assembly – with no robot-specific training. Performance scales with the underlying FM size and strength, demonstrating a promising, scalable path to generalist robot control based on existing multimodal agents. The work highlights an innovative interface design leveraging existing agent skills rather than robot-specific adaptation.

Key findings

• VIA solves a range of six diverse tabletop manipulation tasks zero-shot with off-the-shelf agents, achieving 60% to 88% average success depending on model and prompt (Table 2).
• The strongest configuration (Claude Code Fable 5) achieves 96.7% success on three LIBERO-Goal tasks and 100% on a long-horizon seven-block rainbow assembly task.
• Providing example waypoint demonstrations as text in the prompt boosts performance from 77% to 100% on LIBERO-Goal tasks for CC-Opus, showing learning via prompting.
• Larger and more capable foundation models improve success rates, e.g., upgrading CC-Opus to CC-Fable raises overall success rates from 70% to 88%.
• Agents operate via a minimal set of MCP tools that expose only basic kinematic commands and camera controls, with no privileged robot state or perception primitives.
• VIA’s closed-loop observe-plan-act cycle allows for error recovery by continually taking screenshots after each tool call.
• Average API cost per successful episode varies by model: $4.1 for Codex-5.6-Sol and up to $15.1 for CC-Fable, correlating inversely with tool call efficiency.
• Tasks involving precise placement like T-block remain challenging, with success rates between 10% and 40% for all agents.

Methodology — deep read

1. Threat Model & Assumptions: VIA assumes an adversary-free environment focused on robot control evaluation. The agent does not have privileged access to robot state or specialized training data. Instead, agents are off-the-shelf generalist foundation models capable of computer software interaction through visual inputs. No explicit security adversary is modeled.

2. Data: VIA does not use task-specific training data or fine-tuning. Instead, the agent perceives via screenshots of a browser-based 3D interface built from RGB-D camera streams reconstructing a 3D point cloud workspace, plus raw third-person and wrist camera feeds. Six tabletop tasks (Stack, Turn on stove, Open drawer, Put bowl on plate, Rainbow block assembly, T-block assembly) serve as zero-shot evaluation benchmarks in simulation.

3. Architecture/Algorithm: VIA uses publicly available agentic language models wrapped with Model Context Protocol (MCP) tools exposing observation (screenshots, pose queries), action (target gripper pose commands), camera controls, and execution (move robot, end episode) in human-legible commands. The key novelty is recasting robot control as a visual software use task, enabling the agent to operate the robot manipulator through the interface as if controlling a computer UI.

4. Training Regime: There is no training or fine-tuning of agents on robot tasks. Evaluations involved prompting variants: a minimal prompt specifying only task goals, and a detailed prompt including example waypoint sequences. Testing was performed with multiple seeds (10 per task), using four models: Claude Code Opus 4.8, Claude Code Fable 5, Codex GPT-5.5, and Codex GPT-5.6-Sol.

5. Evaluation Protocol: Success rates were measured for each task and agent configuration over 10 randomized seeds per task. Episodes capped at one-hour wall-clock time for run termination. The number of MCP tool calls and estimated API costs were recorded. Qualitative visual outcomes assessed for the Rainbow task. No adversarial or distribution shift testing performed.

6. Reproducibility: The paper does not explicitly mention public release of code, weights, or datasets within the truncated text. VIA uses third-party agents (Claude Code, Codex) and a custom 3D robot-control UI built on robosuite simulation. MCP tool specifications are documented.

End-to-End Example: For the Open drawer task, VIA’s agent (e.g., CC-Fable) takes a screenshot of the interface, observes the drawer and gripper state, uses MCP tools like gripper_show_rotation_gizmo() and gripper_teleport_via_click() to pose the gripper appropriately near the drawer handle, then calls execute_waypoint to move the physical robot. After observing the result, it re-plans if needed until the drawer is opened sufficiently, indicating task success. This closed-loop cycle repeats for all waypoints necessary to complete the task.

Technical innovations

• Recasting robot manipulation as a visual interface agentic task, leveraging off-the-shelf foundation models designed for computer software use rather than robot-specific fine-tuning.
• Introduction of minimal, human-legible Model Context Protocol (MCP) tools that wrap low-level robot control operations into discrete visual interface actions usable by generalist agents.
• Utilization of a browser-based 3D reconstructed point-cloud UI from RGB-D inputs as a unified perceptual interface that mimics computer software environments familiar to foundation models.
• Demonstration that zero-shot robot control with no privileged perception or skill primitives is possible by offloading visual understanding, planning, and re-planning entirely to strong multimodal agents.

Datasets

• Six manipulation tasks from robosuite simulator: Stack, Turn on stove (LIBERO-Goal), Open drawer (LIBERO-Goal), Put bowl on plate (LIBERO-Goal), Rainbow (custom 7-block task), T-block (BuilderBench re-implementation). Each task evaluated over 10 random seeds.

Baselines vs proposed

• CC-Opus minimal prompt: overall success rate = 70% vs CC-Fable minimal prompt = 88%
• Codex-5.5 minimal prompt: overall success rate = 60% vs Codex-5.6-Sol minimal prompt = 62%
• CC-Opus with detailed prompt (example waypoint demo): 100% on LIBERO-Goal tasks vs 77% with minimal prompt
• Codex-5.5 detailed prompt: no significant improvement from minimal prompt on LIBERO-Goal tasks
• Tool calls per successful episode: CC-Fable uses fewer tool calls than CC-Opus on most tasks despite higher success
• API cost per successful episode: Codex-5.6-Sol at $4.1 avg vs CC-Fable at $15.1 avg (higher capability but more expensive)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.11119.

Fig 1: VIA overview: an off-the-shelf agent controls a robot through a browser-based 3D inter-

Fig 2: One round of the VIA control loop, shown on the open drawer task. Operate (top row,

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 3: The six evaluation tasks. For each task, the top panel shows an example initial state and

Fig 7 (page 4).

Fig 8 (page 4).

Limitations

• Requires state-of-the-art large foundation models for competence, incurring slow and expensive inference.
• Currently limited to quasi-static tabletop tasks; not suitable for dynamic tasks like catching moving objects.
• Performance on tasks requiring very precise manipulation (e.g., T-block) remains limited with success rates 10%-40%.
• No use of privileged simulator or robot state restricts precision and speed but enhances generality.
• No adversarial or distribution shift testing to evaluate robustness.
• Evaluations conducted in simulation with limited preliminary real-world validation.

Open questions / follow-ons

• Can VIA be extended to operate real physical robots effectively, maintaining zero-shot generalization?
• How can more sophisticated low-level robot controllers integrated with VIA improve precision-demanding tasks?
• What are the limits of agent visual perception for manipulation, and can giving agents additional modalities or state feedback improve robustness?
• Can agents learn or improve via reflection and iterative prompting using experience gathered through VIA, effectively enabling online learning without retraining?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, VIA demonstrates a compelling instance of leveraging generalist foundation model agents to control complex physical devices through purely visual interfaces without specialized training. This approach highlights the power of sophisticated visual and reasoning models to operate new modalities by casting control as a visual tool-use problem rather than traditional end-to-end policy learning. The concept of restricting an agent to minimal, human-legible interface tools while offloading perception and planning to a generalist FM could inspire new bot defenses that rely on verifying authentic closed-loop control interactions rather than single token responses. Conversely, attackers might repurpose generalist coding or visual agents for subtle robotic or UI-driven automation beyond conventional scripted bots. Understanding VIA’s methodology may help CAPTCHA designers evolve challenges involving interactive visual or robotic control tasks potentially difficult for adversarial automation agents.

Cite

@article{arxiv2607_11119,
  title={ VIA: Visual Interface Agent for Robot Control },
  author={ Hengyuan Hu and Priya Sundaresan and Jensen Gao and Dorsa Sadigh },
  journal={arXiv preprint arXiv:2607.11119},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.11119}
}

Read the full paper

• arXiv:2607.11119 (abstract)
• arXiv:2607.11119 (PDF)