GROW$^2$: Grounding Which and Where for Robot Tool Use

Source: arXiv:2606.30632 · Published 2026-06-29 · By Yuhong Deng, Yuyao Liu, David Hsu

TL;DR

This paper addresses the challenge of open-world affordance grounding for robot tool use, where a robot must select a suitable tool from a diverse set of objects and precisely localize task-relevant interaction regions without relying on extensive labeled data. The proposed method, GROW2 (GROunding Which and Where), innovatively decomposes the problem into two hierarchical stages: semantic tool and part selection using vision-language models (VLMs) to leverage commonsense knowledge, followed by geometric grounding of selected parts into detailed 3D affordance regions via vision foundation models using a single RGB-D view. This hierarchical approach bypasses the need for end-to-end training on large affordance datasets and enables zero-shot generalization to open-category objects. Extensive experiments on standard 2D and 3D affordance benchmarks (AGD20K and PIAD), a new multi-object tool-selection benchmark (GROW2Bench), simulation with a Franka robot arm, and real-world robot experiments demonstrate that GROW2 consistently outperforms state-of-the-art baselines. It achieves higher task success rates and more precise affordance localization, validating the effectiveness of the part-based decomposition combined with multi-view rendering from a single observation.

Key findings

• GROW2 achieves highest similarity (SIM=0.418) and normalized scanpath saliency (NSS=1.334) on AGD20K unseen split, outperforming baselines like AffordanceLLM (SIM=0.377, NSS=1.070).
• On the PIAD 3D affordance benchmark, GROW2 attains an average IoU (aIoU) of 9.0 and AUC of 74.2, surpassing the previous best GEAL method (aIoU=8.7, AUC=72.5).
• In the newly introduced GROW2Bench multi-object benchmark, GROW2 achieves consistently higher Intersection-over-Union (IoU) across 10 task types versus zero-shot baselines like DINOv3, SD-DINO, and MOKA.
• In robot simulation experiments with a Franka Emika Panda arm over five tasks and 100 scenes, GROW2 attains an average success rate of 86.7%, significantly higher than best baseline UAD (47.0%) and MOKA (33.0%).
• Ablation without multi-view 3D reconstruction drops success from 86.7% to 62.7%, showing the critical role of multi-view fusion in overcoming occlusions.
• Real-world robot experiments on 50 scenes show GROW2 achieves 78.0% average success rate compared to 30.0% of MOKA baseline, confirming transfer from simulation to physical setups.
• Failure analysis indicates largest error source is VLM tool and part selection hallucinations, followed by 3D mesh reconstruction inaccuracies and segmentation errors.
• Runtime analysis reports average scene processing latency of approximately 16.6 seconds due to heavy VLM and vision model queries.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is not explicitly addressed as this is a robotics affordance grounding paper rather than a security paper. The problem involves open-world tool use where the robot receives an RGB-D observation and a natural language instruction describing a task and target object. The system must select a suitable tool object and the relevant parts on tool and target, without prior knowledge or fixed tool-function mappings. There is no training on task-object pairs, so zero-shot generalization is critical.

2. Data: Evaluation uses established affordance benchmarks AGD20K (2D images) and PIAD (3D point clouds) with unseen test splits, a newly created GROW2Bench multi-object benchmark with 10 task types and 50 instances each, as well as simulated and real-world scenes with multiple objects. The paper does not train on these datasets but uses them solely for evaluation.

3. Architecture and Algorithm: GROW2 decomposes the problem hierarchically. First, using a pretrained Vision-Language Model (VLM), the system extracts object names and their part candidates from the RGB image. It uses a Segment Anything Model (SAM3) to segment objects for better VLM hallucination mitigation and crop generation. Then, conditioned on the task instruction and candidate object-part sets, the VLM performs tool selection and selects task-relevant parts on the tool (grasp part pG and action part pA) and target (action part pB).

For geometric grounding, from a single-view RGB-D image, the method performs 3D reconstruction using SAM3D to generate object meshes aligned via ICP to partial point clouds. Multi-view rendering (8 views) of the aligned meshes is performed to expose occluded parts. For each rendered 2D view, SAM3 segments the described parts. If SAM3 fails on complex queries, a VLM predicts bounding boxes which guide segmentation. The multiple 2D masks from different views are back-projected and fused on the mesh surface. DBSCAN clustering removes outliers and retains the major cluster as the 3D affordance region.

4. Training Regime: GROW2 requires no end-to-end training on affordance labels or task-object data. It uses pretrained foundation models: VLMs, SAM3, SAM3D, and ICP-based registration. Few-shot in-context learning in VLM prompt engineering is used for selecting tools and parts, with examples provided.

5. Evaluation Protocol: Metrics include Kullback-Leibler divergence (KLD), similarity (SIM), normalized scanpath saliency (NSS) for AGD20K, and average intersection-over-union (aIoU), area under ROC curve (AUC), mean absolute error (MAE) for PIAD. GROW2Bench uses IoU with human annotations on multi-object scenes. Robotics experiment success rates are assessed in simulation and real with Franka arm executing five manipulation tasks. Baselines include prior SOTA affordance prediction methods and VLM-based zero-shot approaches. Ablation studies test effect of removing multi-view grounding.

6. Reproducibility: Code and model checkpoints are not explicitly stated as released. Dataset splits follow prior public benchmarks. The new GROW2Bench is described but not explicitly public. The approach leverages multiple off-the-shelf foundation models, improving reproducibility. Prompt details and hyperparameters are deferred to appendix.

Concrete Example: Given an instruction “cut the cake,” GROW2 first extracts all candidate objects and parts from the RGB image. The VLM selects the plate as tool and identifies the rim as both grasp and action part on the plate, and the cake part on the target. 3D reconstruction produces meshes for plate and cake, which are multi-view rendered to segment the rim and cake parts across views. The fused 3D segmentation regions provide affordances for grasp and cutting. The robot plans and executes the cut motion using these affordances, enabling improvised plate use as cutting tool.

Technical innovations

• Hierarchical decomposition of affordance grounding into semantic tool and part selection using VLM commonsense reasoning, combined with geometric 3D part grounding using vision foundation models.
• Use of object parts as an intermediate abstraction bridging semantic understanding and geometric grounding to enable zero-shot generalization without large affordance datasets.
• Novel single-view 3D affordance grounding pipeline leveraging mesh reconstruction, multi-view rendering, 2D segmentation with SAM3, and fusion to recover complete, occlusion-robust 3D affordance regions.
• Integration of few-shot in-context learning with VLM prompting to select tools and task-relevant parts in semantic space conditioned on natural language instructions.

Datasets

• AGD20K — Size unstated (standard benchmark) — Public 2D affordance dataset
• PIAD — Size unstated (standard benchmark) — Public 3D affordance dataset
• GROW2Bench — 500 instances (10 tasks × 50 instances) — Newly constructed multi-object affordance benchmark
• Simulation scenes — 100 scenes with 85 object assets and 5 task types — Synthetic via SAPIEN 3
• Real-world scenes — 50 experimental trials over 5 tasks — Physical robot setup

Baselines vs proposed

• Cross-View-AG on AGD20K: SIM=0.285 vs GROW2 SIM=0.418
• LOCATE on AGD20K: KLD=1.405 vs GROW2 KLD=1.506 (higher due to fine-grained predictions)
• GEAL on PIAD: aIoU=8.7 vs GROW2 aIoU=9.0; AUC=72.5 vs GROW2 AUC=74.2
• MOKA on GROW2Bench: average IoU lower across 10 tasks than GROW2
• UAD in simulation (given tool): average success 47.0% vs GROW2 86.7%
• MOKA in simulation: average success 33.0% vs GROW2 86.7%
• GROW2 w/o multi-view: average success 62.7% vs GROW2 full 86.7%
• MOKA real-world success rate: 30.0% vs GROW2 78.0%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.30632.

Fig 1: GROW2 (GROunding Which and Where) decomposes open-world affordance ground-

Fig 2: 3D affordance grounding. Our method consists of two stages: (a) reconstructing and

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 3: Affordance prediction results on GROW2Bench. We evaluate alignment between pre-

Limitations

• The largest failure mode stems from hallucinations or incorrect tool and part selections by the VLM in semantic stage.
• 3D mesh reconstruction inaccuracies (scale or geometry) from single RGB-D observations can lead to poor grounding.
• SAM3 segmentation can fail on complex or ambiguous part queries, requiring fallback to bounding box predictions.
• Robot execution failures occur due to imprecise grasping or motion planning on the predicted affordance regions.
• High computational latency (~16.6 seconds per scene) limits applicability to dynamic or real-time tasks.
• No explicit adversarial or distribution shift robustness evaluation was performed to test in-the-wild generalization beyond evaluated benchmarks.

Open questions / follow-ons

• Can iterative or interactive prompting methods with VLMs reduce hallucination errors in tool and part selection?
• How would performance improve if RGB-D observations are complemented with multi-view camera setups in practice?
• Can the computational efficiency of GROW2 be improved by distilling vision and language foundation models into lightweight affordance-specific architectures?
• How well does GROW2 generalize to entirely novel tasks requiring more complex multi-step tool use or composed affordances beyond single action relationships?

Why it matters for bot defense

This paper's approach of decomposing a complex grounding task into semantic and geometric components demonstrates a promising strategy for bot-defense systems that rely on understanding fine-grained object functionality or interaction reasoning. The use of vision-language models to leverage commonsense knowledge for selecting relevant objects or components in a scene, followed by geometric localization, is analogous to bot-defense scenarios where the system must semantically parse natural language challenges and then verify precise spatial or functional attributes to detect automation. Furthermore, the authors' method of fusing multi-view segmentations into robust 3D regions from a single RGB-D input could inspire more reliable CAPTCHA generation methods that require bots to prove spatial reasoning. Understanding where hallucination errors occur in large language or vision models also informs adversarial robustness strategies. Overall, the research illustrates how combining modern foundation models modularly can enable zero-shot task understanding and grounding with high reliability, a lesson valuable for applying language and vision models in bot-detection and CAPTCHA challenges.

Cite

@article{arxiv2606_30632,
  title={ GROW$^2$: Grounding Which and Where for Robot Tool Use },
  author={ Yuhong Deng and Yuyao Liu and David Hsu },
  journal={arXiv preprint arXiv:2606.30632},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.30632}
}

Read the full paper

• arXiv:2606.30632 (abstract)
• arXiv:2606.30632 (PDF)