VisualClaw: A Real-Time, Personalized Agent for the Physical World

Source: arXiv:2606.16295 · Published 2026-06-15 · By Haoqin Tu, Jianwen Chen, Zijun Wang, Siwei Han, Juncheng Wu, Hardy Chen et al.

TL;DR

VisualClaw presents a novel self-evolving multimodal agent framework designed to efficiently handle streaming video input and adapt its internal skill bank over time without updating VLM weights. It addresses three main deployment gaps: the high cost and latency of dense video frame processing, the static nature of deployed agent scaffolds, and the lack of benchmarks that require genuine multimodal tool use within a workspace. The method combines a CPU-only edge cascade gate to select salient frames for cloud VLM calls, a hybrid hot/cold skill bank injection to limit prompt token cost, and a memory-augmented evolver that distills failures into new or updated skills offline. The result is a large cost reduction (up to -98% API use vs full-frame upload) alongside accuracy improvements averaging +3.85% and up to +15.80% on streaming video-QA datasets using Gemini 3 Flash and GPT-5.2 backbones.

Complementing the method, VisualClawArena, a 200-scenario multimodal agentic benchmark suite, highlights the importance of workspace-style visual grounding by requiring agents to manipulate documents, audio/chat logs, and execute checks in addition to answering questions. On this benchmark, VisualClaw’s learned evolution yields consistent gains of approximately 3 points macro accuracy with Codex and Claude Code backends compared to no-evolution baselines, while also reducing prompt cost. Overall, VisualClaw demonstrates that hybrid encoding and continual skill evolution form a practical framework for real-time, personalized, and cost-effective VLM agent deployment in streaming video contexts and interactive multimodal workspaces.

Key findings

• VisualClaw achieves an average -98.1% per-question VLM API cost relative to full-frame upload across 4 video-QA benchmarks.
• It reduces cost by -25.9% compared to an offline uniform 8-frame baseline while improving accuracy on most datasets.
• On EgoSchema with the Gemini 3 Flash VLM, VisualClaw obtains a +3.85% average accuracy lift and peak +15.80% improvement with full memory-evolution.
• Uniform-8 baseline plus VisualClaw’s full evolution skill updates yields +3.50% to +13.00% absolute accuracy gains across benchmarks including EgoSchema and EgoPlan-Bench.
• At a matched 8-frame budget, cascade frame selection with evolution beats uniform frame sampling by +1.50% to +3.58% on shorter video-QA benchmarks.
• On VisualClawArena, a 200-scenario agentic benchmark, VisualClaw combined with Codex (GPT-5.5) and Claude Code (Sonnet 4.6) backends improves macro accuracy by +2.9% and +3.2%, respectively, over no-evolution baselines.
• VisualClawArena enforces visual grounding through a strict 5-stage data curation pipeline, ensuring >40% steps require video evidence paired with workspace operations.
• Skill bank quality is maintained via token-Jaccard deduplication and per-skill utility tracking, enabling a compact personalized skill set that improves over time without weight updates.

Threat model

The paper does not focus on adversarial threat modeling in the security sense; instead, the threat is operational: excessive latency and cost from dense streaming video uploads and prompt growth, and degradation of agent accuracy from a static skill bank. The adversary is resource constraints and environment variability, not a malicious attacker.

Methodology — deep read

The VisualClaw framework focuses on efficient, adaptive multimodal agent deployment with three main components operating at different timescales.

1. Threat Model & Assumptions: The adversary is implicit in the sense of cost and latency constraints rather than malicious attack, focusing on the deployment challenges of streaming video input and evolving skill banks under frozen VLM weights. The agent cannot update model weights but can grow and adapt its skill and memory banks.

2. Data: VisualClaw is evaluated on four video-QA benchmarks with different characteristics — EgoSchema (500 long egocentric clips), EgoPlan-Bench (923 single-frame ego planning), Video-MME long (900+ longer clips), and NextQA (1000 short clips), plus the newly curated VisualClawArena benchmark consisting of 200 multimodal agentic scenarios built from existing datasets (Indoor/VSI, EgoSchema, QVHighlights). The Arena scenarios involve video, documents, and executable workspace actions with 3,106 total instruction rounds. Videos are sampled at 1 FPS for streaming evaluation.

3. Architecture / Algorithms: VisualClaw defines the agent M=(θ, S, Mv, G) where θ are frozen cloud VLM weights, S is a skill bank of language-level reusable skills, Mv is an episodic memory store indexed by dense semantic embeddings, and G is an edge-side frame filter gate.

• The Per-frame module is a three-stage CPU-only cascade gate G that inspects arriving frames using perceptual hashing, a lightweight 128-dim feature vector (HSV histograms, luminance, texture), and an adaptive threshold to classify frames as MAJOR (keyframe), MINOR (scene-updater), or SKIP. Only MAJOR frames are uploaded to the VLM, drastically reducing bandwidth.

• The Per-question module implements hybrid hot/cold skill bank injection. Top-k retrieved skills relevant to the current query (using sentence-transformer embeddings) form the hot set fully inlined in the prompt, while the rest remain a cold catalog only referenced by name and short description, bounding prompt token cost.

• The Per-session module triggers a memory-augmented evolver E offline every N_evo=15 failures, which uses retrieved high-confidence memories Mv relevant to failure instances D_fail to generate skill updates ∆S. Two evolver input variants are studied: simple concatenation of memory and failures (Cat.), and a guided input that instructs the evolver to extract reusable skills while avoiding scenario-specific detail (Guide). Post-processing filters deduplicate skills by token-Jaccard similarity and track per-skill utility to prune underperforming skills.

4. Training regime: There are no model weight updates for the cloud VLMs which remain frozen. The only learning is through the offline evolver generating skill bank updates via calling an LLM (Claude Haiku 4.5) on batches of failure episodes. Memory embeddings use all-MiniLM-L6-v2. Videos are processed at 1 FPS. Evaluation uses multiple seeds and hyperparameters for cascaded thresholds and top-k sizes, but exact hyperparameter sweep details are not specified.

5. Evaluation Protocol: Quantitative evaluation on 4 video-QA datasets and VisualClawArena. Metrics are accuracy (top-1 multiple choice or final scenario pass macro averages), evaluated under different settings including Plain (no evolution), Seed (bootstrap skill bank), +Evolve (skill bank updates), +SkillMemCat (answer-time memory concatenation), and FullEvo variants (Cat. and Guide) that use memory-augmented skill evolution. Costs are measured by number of VLM API calls and prompt token injection. Ablations analyze frame gating efficiency, skill bank growth, memory guidance, and cross-backbone generalization (Gemini 3 Flash vs GPT-5.2). VisualClawArena evaluation involves agentic backends (Codex, Claude Code) executing multimodal tool-using steps, scored via executable checks.

6. Reproducibility and Code: The paper mentions a project page but does not specify public code release or weights. The experiments rely on proprietary LLMs (Gemini, GPT-5.x, Claude). VisualClawArena scenarios are newly curated; source video datasets are public but the transformed agentic data and tooling are not clearly described as open.

Concrete example: For a streaming 3-minute video question in EgoSchema, VisualClaw’s cascade gate selects ~8 major keyframes by edge CPU-only processing, which are uploaded with hot skills injected from a dynamically evolving skill bank seeded with 12 initial skills and expanded via offline evolver every 15 failures. The agent retrieves relevant episodic memories and injects them as guided input to the evolver prompting generalized skill creation. The updated skill bank leads to +15.8% accuracy lift over plain baseline on this dataset while reducing the VLM API cost by over 98%.

Technical innovations

• A CPU-only cascaded video frame encoding gate running online at the edge that selects key frames based on lightweight perceptual hashing and adaptive scene-change thresholds without any future frame lookahead.
• A hybrid hot/cold skill bank prompting scheme injecting top-k fully expanded skills inline while exposing remaining skills as a compressed named catalog to bound prompt token cost independently of skill bank size.
• A memory-augmented offline skill evolver that uses retrieved high-confidence failure episodes as guided context to adapt and evolve the skill bank without updating frozen VLM weights.
• VisualClawArena, a novel 200-scenario multimodal agentic benchmark requiring use of visual evidence alongside document editing and executable workspace checks, curated with a strict five-stage pipeline enforcing visual grounding.

Datasets

• EgoSchema — 500 instances, average 3-minute clips — public
• EgoPlan-Bench — 923 entries, single-frame egocentric planning task — public
• Video-MME long — 900 long video clips, multiple tasks — public
• NextQA — 1000 short (~30s) YouTube clips for causal/temporal reasoning — public
• VisualClawArena — 200 curated multimodal agentic scenarios with 3,106 steps total, sourced from Indoor/VSI (100), EgoSchema (50), QVHighlights (50) — dataset newly curated, not publicly released

Baselines vs proposed

• Full-frame upload: API cost baseline, VisualClaw achieves -98.1% cost vs this baseline across 4 video-QA datasets
• Uniform-8 offline uniform frame sampling: VisualClaw reduces cost by -25.9% and improves accuracy by average +3.85%
• EgoSchema Gemini 3 Flash Plain: 52.6%, Seed: 67.2%, FullEvo (Guide): 68.4% (streaming), Uniform-8 Plain: 60.6%, Uniform-8 + FullEvo (Guide): 73.6%
• VisualClawArena Codex baseline (no evolution): ~51.4% macro accuracy, VisualClaw + FullEvo (Guide): 54.3% (+2.9%)
• VisualClawArena Claude Code baseline: ~49.0%, VisualClaw + FullEvo (Guide): 52.2% (+3.2%)
• EgoPlan-Bench Uniform-8 Plain: 37.96%, Cascade-fill + FullEvo: 52.06% (+14.10%)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.16295.

Fig 1: VisualClaw can efficiently encode streaming video in real scenarios and produce personal-

Fig 2 (page 1).

Fig 3 (page 1).

Fig 2: VisualClaw pipeline. A three-timescale system with hybrid encoding and meta-evolve. An

Fig 3: VisualClawArena curation pipeline. We choose videos from three different sources and

Fig 4: A complete example from VisualClawArena with an video clip, scenario-related files, and

Fig 6: Case studies of VisualClaw. Case 1 (EgoSchema, +14.60%): single MAJOR keyframe

Fig 7: Two further VisualClaw wins, including a cross-VLM transfer. Case 3 (EgoPlan-Bench,

Limitations

• The skill evolution relies on offline LLM calls and a fixed failure batch size, limiting real-time adaptivity.
• The VisualClawArena benchmark scenarios and tooling are proprietary and not publicly released, limiting reproducibility.
• Evaluations focus on two specific VLM backbones (Gemini 3 Flash, GPT-5.2) and two agent backends (Codex, Claude Code), with unclear generality to other models.
• The cascade gating thresholds are heuristically tuned and may not generalize across all video domains or frame rates.
• No direct adversarial robustness evaluation or tests against deliberate evasion of frame gating or memory manipulation are presented.
• The approach currently assumes high-quality memory retrieval embeddings and relevance ranking, which may degrade in noisier settings.

Open questions / follow-ons

• Can skill evolution be performed online or in real time rather than in batches after multiple failures?
• How would VisualClaw perform with other large multimodal transformers or smaller edge models?
• What attack vectors exist if adversaries attempt to poison the skill or memory bank to degrade agent performance?
• Can the cascade thresholds and embedding retrieval be learned end-to-end for further efficiency gains?

Why it matters for bot defense

VisualClaw’s hybrid encoding and memory-augmented skill evolution present a meaningful direction for bot-defense and CAPTCHA practitioners facing real-time video or multimodal challenges. The edge-side cascade gating dramatically reduces computation and bandwidth costs, which is critical for live defenses involving video streams or rich sensory input. Additionally, evolving the agent’s internal skill set from past failures without model retraining could inform continuous adaptation strategies in CAPTCHA solvers or bot detectors that must learn from attack patterns iteratively. VisualClawArena’s emphasis on workspace-level multimodal grounding also aligns with realistic bot tasks requiring executing multi-step operations, not just single-shot Q&A, highlighting the necessity for benchmarks reflecting agent persistence and adaptation. However, the requirement for offline evolution and reliance on non-public datasets means practitioners must consider the complexity of implementing similar frameworks in real-world settings.

Cite

@article{arxiv2606_16295,
  title={ VisualClaw: A Real-Time, Personalized Agent for the Physical World },
  author={ Haoqin Tu and Jianwen Chen and Zijun Wang and Siwei Han and Juncheng Wu and Hardy Chen and Haonian Ji and Kaiwen Xiong and Jiaqi Liu and Peng Xia and Jieru Mei and Hongliang Fei and Jason Eshraghian and Zeyu Zheng and Yuyin Zhou and Huaxiu Yao and Cihang Xie },
  journal={arXiv preprint arXiv:2606.16295},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.16295}
}

Read the full paper

• arXiv:2606.16295 (abstract)
• arXiv:2606.16295 (PDF)