Agent-Computer Observation Interfaces Enable Dynamic Computer Use

Source: arXiv:2606.29472 · Published 2026-06-28 · By Bojie Li, Noah Shi

TL;DR

This paper addresses a critical shortcoming in computer-use (CU) agents: their limited observation interfaces, which currently consist of a single static screenshot every 3–5 seconds and no audio input. This observation constraint renders CU agents effectively blind and deaf to dynamic content such as video, animations, transient UI events, and spoken instructions. The authors propose the Agent-Computer Observation Interface (AOI), a model-agnostic perception layer that decouples continuous observation from discrete action steps. AOI integrates three gated components: inter-step keyframe capture, volume-gated audio transcription, and CU-model-generated visual narration that persists as text. AOI produces near-zero overhead on static content but enables rich dynamic and multimodal inputs when relevant.

The authors validate AOI on a newly introduced benchmark, DynaCU-Bench, comprising 100 browser-based tasks that require dynamic visual and/or audio perception plus 50 static controls. Evaluated on eight CU models ranging from 7 billion to frontier-scale parameters, AOI consistently lifts task success rates by 17 to 48 percentage points without any retraining, turning near-impossible screenshot-only tasks into largely solved problems. The largest gains occur on audio-reliant tasks (e.g., meetings, podcasts), where AOI agents solve every task by capturing and transcribing speech. A detailed ablation shows that most of AOI's value comes from persistent textual narration rather than selecting keyframes or audio alone. AOI thus unlocks a previously neglected design axis for CU agents, enabling multimodal, continuous observation while preserving existing model architectures.

Key findings

• On DynaCU-Bench's 100 dynamic tasks, AOI improves task success by +17 to +48 percentage points over screenshot baselines across 8 CU models, with zero retraining.
• Claude Sonnet 4.6 gains +44 pp (38%→82%) and Gemini 2.5 Flash gains +48 pp (21%→69%), the largest absolute improvements.
• Audio-enabled families (Podcast, Phone, Meeting) go from near-zero success to nearly solved with AOI, capturing speech crucial for these tasks.
• Visual-temporal families (Screencast, Carousel, Transient) also show consistent positive gains; Dashboard benefits less; Games regress due to latency sensitivity.
• Ablation shows keyframe selection algorithms perform similarly; most gains come from CU-model-generated persistent textual narration.
• AOI's modular gated design produces near-zero overhead on static silent content, confirmed by near-equal performance on 50-task static controls.
• Open-source EvoCUA-32B (32B params) goes from 18% to 55% success (+37 pp) with AOI, rivaling larger closed-source models.
• The newest Gemini 3 Flash model shows only +9 pp improvement due to offsetting component effects, necessitating per-model component tuning.

Threat model

Not a security-focused paper; adversary is not explicitly modeled. The goal is to expand CU agents’ sensory input capabilities beyond sparse screenshots and muted observation intervals, enabling perception of transient visual and audio computer-use cues inaccessible under current observation interfaces.

Methodology — deep read

The authors propose AOI to expand a CU agent's observation space from sparse screenshots to continuous, gated multimodal perception without retraining the underlying CU models.

Threat Model & Assumptions: The adversary concept is implicit; the goal instead is to overcome the CU agents' blind spots—the inability to perceive transient visual and audio content occurring between static screenshots. The AOI does not defend against adversarial attacks but aims to improve dynamic and audio content awareness.

Data: The authors construct DynaCU-Bench, a benchmark with 100 dynamic browser tasks requiring temporal visual/audio perception and 50 static HTML control tasks. They design tasks spanning audio perception (AUD), visual-temporal perception (VIS), and real-time interaction (INT) across ten families, stratified by difficulty (easy, medium, hard). The environment is a headless Chromium browser with virtual audio devices to replay synthesized speech.

Architecture / Algorithm: AOI is a modular Python layer placed between the environment and any existing image-based CU model. AOI continuously observes screen and audio via three main components:

1. Inter-step keyframe capture samples the screen ∼3 Hz and applies a fast pixel-change gate (<1% pixel difference) to select up to 5 keyframes per step. Several algorithms (uniform, pixel diff, CLIP semantic filtering) are evaluated, with pixel gating sufficient.
2. Volume-gated audio observer computes RMS energy over audio buffer per step and activates Whisper large-v3 ASR on non-silent intervals to produce text transcripts.
3. Visual narration uses the CU model to generate brief textual descriptions of newly observed keyframe content at each step, which persist in the trajectory after images are pruned.

Training Regime: No retraining is performed; the CU models are evaluated as-is, with AOI feeding them augmented observations. Models use vendor-recommended decoding parameters.

Evaluation Protocol: Evaluation uses DynaCU-Bench tasks run in an automated headless browser+audio system paired with CU models (closed and open-source). Success is determined mostly by deterministic state checks, supplemented by an LLM rubric for some tasks. Task timeouts and max steps (15) bound runs. Paired McNemar exact tests evaluate significance of improvements over baselines.

Reproducibility: Code and benchmark datasets are publicly released at https://github.com/19PINE-AI/aoi. CU models include open-source and closed-source variants; closed weights are not disclosed but recipes and APIs are used.

Example: On a meeting task, AOI enables continuous audio transcription of spoken conference information unavailable in sparse screenshots. The CU agent receives both the transcript and persistent visual narration of slides, allowing it to answer temporal questions and react in fewer steps compared to a baseline screenshot-only agent that misses crucial spoken content.

Technical innovations

• Decoupling observation from action in CU agents by introducing a model-agnostic continuous perception layer (AOI) that integrates adaptive keyframe capture, volume-gated audio transcription, and persistent visual narration text.
• Use of fast, low-overhead gating mechanisms (<1 ms) to selectively process screen and audio data only when significant changes or sound occur, minimizing cost on static or silent content.
• Leveraging the CU model itself to generate persistent textual visual narrations that accumulate in the observation history to overcome the limited visual memory due to keyframe pruning.
• Introduction of DynaCU-Bench, a new benchmark specifically targeting dynamic browser-based tasks requiring temporal and multimodal perception not solvable from screenshots alone.

Datasets

• DynaCU-Bench — 150 tasks (100 dynamic + 50 static) — synthetic headless Chromium browser environment with virtual audio devices

Baselines vs proposed

• Claude Sonnet 4.6: standard screenshot baseline = 38%, AOI full = 82% (+44 pp)
• GPT-5.4: standard = 37%, AOI full = 57% (+20 pp)
• Gemini 2.5 Flash: standard = 21%, AOI full = 69% (+48 pp)
• Grok-4: standard = 4%, AOI full = 25% (+21 pp)
• EvoCUA-32B: standard = 18%, AOI full = 55% (+37 pp)
• Fara-7B: standard = 17%, AOI full = 34% (+17 pp)
• Gemini 3 Flash: standard = 36%, AOI full = 45% (+9 pp; not statistically significant)
• Qwen3-VL-235B-A22B-Instruct: +42 pp (22→64%) AOI gain
• Qwen3-VL-30B-A3B-Instruct: +24 pp (18→42%) AOI gain

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.29472.

Fig 5: What the agent actually sees: real frames and narrations from a recorded benchmark run

Fig 2 (page 7).

Fig 3 (page 7).

Limitations

• AOI adds per-step observation latency, which can harm performance on latency-sensitive interactive tasks such as games.
• Audio observation currently transcribes speech only; non-speech audio events (e.g., notifications) are gated but ignored in text output, limiting audio modality completeness.
• Keyframe selection algorithms (pixel-based, semantic) show little difference, indicating that AOI’s design may be further optimized.
• Latest CU models (e.g., Gemini 3 Flash) require per-model tuning of AOI components as blanket configurations can degrade performance.
• DynaCU-Bench tasks lack some real-world complexity, e.g., adversarial observation evasion or noisy audio conditions.
• No direct adversarial robustness evaluation; primary focus is improved observation rather than security.

Open questions / follow-ons

• How can AOI be extended to incorporate non-speech audio event interpretation (e.g., notification sounds, error beeps) for richer multimodal awareness?
• What are optimal per-model AOI component configurations for different CU architectures, and can automatic tuning methods be developed?
• How does AOI perform under real-world conditions with noisy audio, variable frame rates, or adversarial UI manipulations?
• Can AOI principles be adapted for latency-critical interactive computer use, such as gaming, where observation overhead impacts performance?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this work highlights a new design axis beyond action interface improvements: augmenting observation with continuous, multimodal inputs. AOI demonstrates that bot or agent perception can be substantially enhanced by decoupling sensing from discrete action steps, enabling richer dynamic and audio content capture without retraining models. This suggests defense systems or CAPTCHA challenges relying on transient UI events or audio cues should consider agents that incorporate similar observation layers to improve robustness and interaction fidelity. Additionally, AOI’s gating mechanisms exemplify how to preserve efficiency on static content while enabling dynamic awareness, a principle applicable to balancing security responsiveness and resource use in live bot detection systems.

Cite

@article{arxiv2606_29472,
  title={ Agent-Computer Observation Interfaces Enable Dynamic Computer Use },
  author={ Bojie Li and Noah Shi },
  journal={arXiv preprint arXiv:2606.29472},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.29472}
}

Read the full paper

• arXiv:2606.29472 (abstract)
• arXiv:2606.29472 (PDF)