HumanCLAW: Can Vision-Language Models Act Through a Body?

Source: arXiv:2607.27180 · Published 2026-07-29 · By Siyao Li, Jiawei Gu, Shuai Liu, Kairui Hu, Zekun Li, Linjie Li et al.

TL;DR

HumanCLAW addresses the fundamental challenge of evaluating whether vision-language models (VLMs) can exhibit action intelligence by controlling a full-body humanoid in physically realistic environments. Prior work either uses abstract discrete actions or learns policies tightly coupled to embodiment motor control, making it impossible to isolate decision-making quality from low-level motion execution failures. HumanCLAW introduces a novel evaluation framework that decouples the VLM's high-level atomic skill decision from physical motor control by using a skill-conditioned motion generator and a half-physics simulator. This setup preserves realistic physical consequences such as gravity, collision, and object interaction, yet factors out balance and motor-tracking failures as confounders.

Based on this framework, the authors curate HumanCLAW-Bench, a large benchmark with 1,218 egocentric find-navigate-interact episodes across 41 indoor scenes, where tasks require a long-horizon sequence of atomic skill decisions. Evaluating nine state-of-the-art VLMs on this benchmark reveals that none can reliably solve it, with the best model achieving only 16.8% success. Importantly, perception of targets is not the limiting factor; rather, the gap lies in embodied self-awareness, where VLMs fail to maintain an accurate representation of the agent’s own body state and spatial relation to the environment. These results highlight current VLMs’ inability to combine spatial understanding with closed-loop whole-body decision-making in complex physical settings.

Key findings

• HumanCLAW-Bench contains 1,218 long-horizon, egocentric find-navigate-interact episodes spanning 41 indoor houses.
• The best performing VLM model reaches only 16.8% overall success rate on completing full find–navigate–interact tasks.
• Target object recognition performs near the theoretical visibility upper bound, with no more than 5 percentage points difference.
• Failures arise primarily after perception: VLMs cannot reliably determine their body location relative to the scene, arrival at goals, or detect collisions.
• Use of a half-physics simulator factors out failures from motor control and balance, allowing attribution of errors specifically to the decision-making level.
• A skill-conditioned motion generator realizes each atomic skill as a realistic 0.5s continuous 100+ DoF full-body motion chunk.
• Skill verification rejects spatially unsafe or premature skill proposals, improving decision consistency despite VLM hallucinations over longer rollouts.
• A progressive high-to-mid-to-low reasoning scaffold guides the VLM to ground long-horizon goals into stepwise atomic skill decisions.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is not the focus here as this is not a security paper, but the setup assumes an off-the-shelf, frozen vision-language model (VLM) controls the agent. The model has access only to egocentric RGB observations and a textual history of past observations and actions. Motor execution is decoupled to isolate the decision-making capability from low-level biomechanical and balance issues.

2. Data: HumanCLAW-Bench is collected over 1,218 episodes distributed across 41 realistic indoor house scenes sourced from the AI Habitat environment with Bullet physics. Episodes require finding, navigating to, and interacting with specified target objects. episodes are labeled with ground-truth object locations, agent poses, and success criteria. There are staged difficulty levels varying geodesic distance, obstacle density, and episode complexity.

3. Architecture / Algorithm: The decision-making module harnesses an off-the-shelf, open-ended VLM prompted in a contextual scaffold. Each step, the VLM receives a prompt containing identity, task instruction, available atomic skills, explicit ego-centric visual state assessments (visible objects, target visibility, surroundings), and recent history encoding. The model performs progressive reasoning: from high-level task understanding to mid-level dynamic goals to low-level atomic skill selection with continuous parameters. A skill verifier module uses compact, skill-specific rule checks to accept or reject proposed skill commands based on spatial feasibility and reasoning correctness.

To execute the skill commands, a skill-conditioned motion generator uses a diffusion transformer (DiT) trained on AMASS human motion data. The base DiT produces smooth future motion tokens given recent pose history. Controlled by skill-specific ControlNet adapters, it generates physically realistic, parameterized 0.5s motion chunks corresponding faithfully to the atomic skill requested.

A half-physics simulator runs at 120Hz with a Bullet engine, simulating rigid body interactions including gravity, collisions, friction, and movable objects. Instead of joint torques, the humanoid is driven by equivalent kinematic velocities derived from the motion generator output. This ensures realistic environmental consequences while abstracting away failures from low-level motor control.

4. Training regime: The base DiT is trained with flow matching on AMASS motion clips segmented into 20 frames at 30Hz; the last 15 frames predict future motion velocities from noisy inputs. Skill ControlNet adapters are trained separately on filtered skill-specific clips with an AdamW optimizer, batch size 2048, and learning rate 3e-4 for ~0.5 to 1.5 million steps. The base model weights are frozen during adapter training.

5. Evaluation protocol: Evaluation measures the overall episode success (completion of find-navigate-interact tasks within time), target object perception accuracy, skill selection correctness, and failure analysis attributing errors to perception versus action intelligence. Nine state-of-the-art VLMs are tested in zero-shot fashion with no action training. Spatial metrics include geodesic distance to targets, obstacle encounters, and collision detection.

6. Reproducibility: The project page releases benchmarks and code bases. The dataset (HumanCLAW-Bench) and motion models are constructed from public sources (AMASS, Habitat). Full details and prompt scaffolds are documented.

Example Workflow: At timestep t, the harnessed VLM receives egocentric RGB frame ot and text history ht, then outputs a JSON skill command <st, ct> from the atomic skill set (e.g., walk, turn, sit with parameters). The skill verifier checks spatial feasibility; if rejected, a corrected skill is chosen. The skill-conditioned motion generator creates a 0.5s continuous full-body motion chunk ˆxfut realizing the skill. The half-physics simulator executes the motion under physical constraints, updating the scene state and rendering the next observation ot+1 for time t+1. The closed loop continues until task completion or failure.

Technical innovations

• Decoupling high-level VLM atomic skill decision-making from low-level motor control via a skill-conditioned motion generator and half-physics simulator.
• A progressive high-to-mid-to-low reasoning scaffold that elicits explicit egocentric spatial state and stepwise skill decisions from an off-the-shelf VLM.
• Skill-conditioned motion generation using a frozen motion base DiT model with plug-and-play ControlNet adapters for parameterized whole-body motion primitives.
• A half-physics simulation environment that simulates physical interaction and contact with the environment but abstracts motor execution to isolate action intelligence evaluation.

Datasets

• HumanCLAW-Bench — 1,218 episodes — constructed on AI Habitat 3D indoor scenes with Bullet physics engine
• AMASS — 40+ hours of mocap data — public large-scale motion capture dataset for skill-conditioned motion generation

Baselines vs proposed

• State-of-the-art vision-language models (9 variants) zero-shot: task success rate ranges from near 0% up to 16.8% vs HumanCLAW framework exposing full-body embodied closed-loop control
• Target recognition accuracy for best model: within 5 percentage points of theoretical visibility upper bound, indicating perception is not the bottleneck

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.27180.

Fig 1: Where does action intelligence live? Three regimes of action along increasing physical embodiment: (a) action

Fig 2: HumanCLAW closed-loop framework. A harnessed off-the-shelf VLM maps each egocentric observation to a

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 2).

Limitations

• HumanCLAW evaluates only zero-shot off-the-shelf VLMs without policy fine-tuning, leaving open how specialized training could improve action intelligence.
• The skill-conditioned motion generation relies on AMASS mocap data which may not fully represent all physically possible motions or complex interactions.
• The half-physics simulator abstracts balance and motor-skills which can be critical in real embodied agents; results may not transfer directly to fully dynamic control.
• The benchmark focuses on indoor find-navigate-interact tasks, limiting generalization to outdoor or multi-agent settings.
• The evaluation does not include adversarial or real-world physical noise perturbations.
• No explicit memory or neural belief-state module is incorporated, which may limit the model’s embodied self-awareness capabilities.

Open questions / follow-ons

• How can VLMs be extended or fine-tuned to improve embodied self-awareness and maintain an internal model of body state during long-horizon tasks?
• Can integrating explicit spatial memory or learned belief states boost closed-loop decision-making in humanoid settings?
• How well does the decoupling approach generalize to fully dynamic motor control where balance and physics constraints become stricter?
• What additional atomic skills or richer motion primitives are required to scale to more complex physical interactions?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, HumanCLAW highlights the critical gap between perception and action intelligence in embodied AI systems. Analogously, CAPTCHA systems relying on human-like embodied reasoning should consider whether models can truly possess the self-awareness to track their own state and interact physically in a world, beyond recognizing visual cues. HumanCLAW's approach to decoupling decision-making from low-level motion control may inspire improved evaluation methods for embodied AI robustness and bot detection. Furthermore, the progressive reasoning scaffold and skill verification approach suggest useful paradigms for structuring complex human-like tasks into atomic verifiable steps, aiding reliable detection of automation versus genuine human-like agency.

Cite

@article{arxiv2607_27180,
  title={ HumanCLAW: Can Vision-Language Models Act Through a Body? },
  author={ Siyao Li and Jiawei Gu and Shuai Liu and Kairui Hu and Zekun Li and Linjie Li and Chengcheng Tang and Po-Chen Wu and Ivan Shugurov and Lingni Ma and Michael Zollhoefer and Sizhe An and Abhay Mittal and Amy Zhao and Ranjay Krishna and Manling Li and Ziwei Liu and Chuan Guo },
  journal={arXiv preprint arXiv:2607.27180},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.27180}
}

Read the full paper

• arXiv:2607.27180 (abstract)
• arXiv:2607.27180 (PDF)