VTLoc: Learning-based Tactile Contact Localization in Visual Point Clouds

Source: arXiv:2607.16146 · Published 2026-07-17 · By Zhiyuan Wu, Zhuo Chen, Shan Luo

TL;DR

This paper tackles the problem of localizing tactile contacts on an object's surface by jointly leveraging tactile sensor readings and visual 3D geometry in the form of point clouds. Accurate tactile contact localization is a challenging cross-modal problem due to the differing spatial dimensions and the need to align tactile information with global object geometry. The authors propose VTLoc, a novel learning framework combining a geometric multi-modal alignment (GMA) module that reconstructs a pseudo-point cloud from fused visual and tactile features and aligns it with the original point cloud using Chamfer distance, and an iterative localizing updater (ILU) that refines contact location predictions across multiple steps using a GRU-based recurrent regression. VTLoc is evaluated on a new large-scale benchmark with 100 real-world objects and over 30 tactile contacts per object, collected with a GelSight sensor and scanned with a high-accuracy 3D scanner. Quantitatively, VTLoc consistently outperforms prior baselines such as Point Filtering, MCR, and MidasTouch on various metrics including normalized distance error, Top-1 and Top-5 accuracy, and mean probability error, with gains especially strong on objects with non-uniform surface curvature. Ablations confirm the importance of both GMA and ILU modules, as well as using surface normals. The model also achieves better multi-contact sequential localization and generalizes reasonably to unknown objects. Qualitative results demonstrate robust and interpretable contact probability heatmaps even for geometrically complex objects.

Key findings

• VTLoc reduces normalized distance error (ND) by 14.87% and improves Top-5 accuracy by 30.19% compared to the Point Filtering baseline on the non-uniform object subset.
• Incorporating surface normals into the point cloud input further improves VTLoc’s ND by 0.74%, Top-1 accuracy by 6.91%, Top-5 accuracy by 1.88%, and mean probability error (MPE) by 7.16% compared to using coordinates alone on the non-uniform subset.
• VTLoc achieves a Top-1 accuracy of 44.65% and Top-5 accuracy of 86.16% with GMA and ILU, outperforming variants without GMA or iterative updates by over 10% absolute in Top-1 accuracy.
• Increasing the number of ILU iterations to 16 optimizes performance, reducing ND from 22.97% to 20.17%, and improving Top-1 accuracy from 29.56% to 44.65%. Excessive iterations reduce performance due to overfitting.
• On the uniform curvature subset, VTLoc reduces ND by 12.11% and improves Top-5 accuracy by 25.35% over Point Filtering but performance is lower than on the non-uniform subset due to geometric ambiguity.
• VTLoc combined with multi-contact sequential observations reduces ND from 20.17% to 4.85% on non-uniform objects, outperforming MidasTouch’s 8.76% ND in the multi-contact setup.
• VTLoc generalizes better to unknown objects with improvements of over 6% Top-1 accuracy and a lower mean probability error (0.1883) compared to baselines.
• Applying rotational symmetry priors enables VTLoc to represent multiple plausible contacts on symmetric objects, addressing intrinsic ambiguity.

Threat model

The adversary is an environment or scenario causing potential ambiguity in contact localization by providing similar tactile signals for multiple different locations on an object's surface, as occurs naturally with symmetric or smooth geometries. The system assumes tactile images and point cloud geometry as inputs, with no adversarial manipulations of sensor data, calibration, or attack strategies. The adversary cannot alter sensor hardware or perturb the modalities beyond natural variations. The threat focuses on ambiguity arising from geometric and sensory modality limitations.

Methodology — deep read

1. Threat Model & Assumptions: The adversary scenario is primarily a robot interacting with objects using vision-based tactile sensors and RGB-D cameras. The model assumes access to tactile images from a GelSight sensor and a 3D point cloud representing the object geometry. The adversary cannot control the sensor calibration or distort sensor readings but faces geometric ambiguities especially on smooth or symmetric surfaces.

2. Data: The dataset is based on ObjectFolder Real, augmented with 100 everyday objects scanned with an EinScan Pro HD 3D scanner producing high-accuracy meshes and point clouds. Each object has 30-50 selected contact points where a Franka Emika Panda robot equipped with a GelSight tactile sensor performs tactile data collection via controlled probing motions. The point clouds have 1024 points per object sampled along with surface normals. Data splits use a 7:3 ratio for train/test at the data level, and unknown object splits are done at the object level for generalization tests.

3. Architecture & Algorithm: VTLoc consists of two main modules:

   • Tactile encoder: ResNet-18 followed by flattening MLP encodes tactile images into feature vector Ft.
   • Visual encoder: PointNet++ encodes point cloud with coordinates and optionally normals into feature vector Fp.
   • Fusion: The tactile and visual features are concatenated into Ff = [Ft, Fp].
   • Geometric Multi-Modal Alignment (GMA): uses learned weights to reconstruct a pseudo-point cloud Pd from fused features. An MLP decodes each vector to a 3D point with normal. The Chamfer distance between Pd and the input point cloud P is minimized during training to enforce geometric consistency.
   • Iterative Localizing Updater (ILU): a recurrent GRU-based module predicts iterative refinements {ĉ1,...,ĉN} starting from zero. The hidden state is updated using tactile and visual features, with MLP decoding each hidden state to a delta update for the contact location prediction. The final ĉN is the predicted contact point.
   • Contact probability heatmap is generated over the point cloud by ranking distances from predicted contact to discrete candidate contacts and computing weights with exponential kernel.

4. Training Regime: The model optimizes a combined loss L = Lseq + λLrecon with λ=1.

   • Lrecon is Chamfer Distance between Pd and P.
   • Lseq is weighted sum of positional errors across ILU iterations, with weights exponentially increasing. Iterations N is tuned with best performance at N=16. Batch size, optimizer, epochs, and hardware details are not specified in the source text.

5. Evaluation: Metrics include millimeter error, normalized distance error (scale-normalized), Top-1 and Top-5 accuracy against discrete contact candidates, and mean probability error (MPE) against soft contact heatmaps. Baselines include Point Filtering, MCR, and MidasTouch. Experiments evaluate on ‘‘non-uniform’’ and ‘‘uniform’’ curvature subsets and unknown object splits. Ablation studies assess the impact of GMA, ILU iterations, and modality inputs. Multi-contact sequential localization uses a probabilistic scoring head and belief update from prior multi-touch data.

6. Reproducibility: Code and data are released at the authors’ website (https://georgewuzy.github.io/vtloc-website/). Dataset ObjectFolder Real is public. Exact training hyperparameters and seeds are not specified.

Concrete Example: For a tactile image from touching a swan figurine’s surface, the tactile encoder processes the RGB tactile image, PointNet++ encodes the 3D object point cloud with normals, fused features are passed to GMA to reconstruct a pseudo-point cloud aligned with the original via Chamfer loss enforcing spatial consistency. The ILU iteratively refines the predicted contact coordinates starting from zero to a final estimate mapped to the nearest candidate point(s). The output is a contact probability heatmap showing likely touch locations on the swan’s surface, matching tactile impressions to geometry.

Technical innovations

• A geometric multi-modal alignment (GMA) module that reconstructs a pseudo-point cloud from fused visual and tactile features and enforces spatial consistency by minimizing Chamfer distance with the input point cloud.
• An iterative localizing updater (ILU) module using a GRU-based recurrent regression to progressively refine contact location predictions over multiple iterations.
• First large-scale visual-tactile contact localization benchmark of 100 real-world objects, with extensive real tactile data collection paired with high-fidelity 3D scanned point clouds.
• Demonstration that fusing tactile images with 3D point clouds augmented by surface normals improves contact localization accuracy, especially on non-uniform surface geometries, without reliance on dense codebooks or randomized sensor poses.

Datasets

• ObjectFolder Real — 100 real-world daily objects — publicly available dataset extended with additional data collection
• Tactile images collected with GelSight sensor — multiple contact points per object — captured via Franka Emika Panda robot arm
• High-accuracy 3D meshes and point clouds obtained with EinScan Pro HD 2020 handheld 3D scanner

Baselines vs proposed

• Point Filtering (non-uniform subset): ND = 24.02% vs VTLoc (coords+normals): 20.17%
• MCR (non-uniform subset): Top-5 Acc = 78.62% vs VTLoc (coords+normals): 86.16%
• MidasTouch (uniform subset): Top-5 Acc = 55.20% vs VTLoc (coords): 64.36%
• MidasTouch (multi-contact non-uniform): ND = 8.76% vs VTLoc (multi-contact): 4.85%
• VTLoc without GMA (coords+normals): Top-1 Acc = 33.65% vs VTLoc with GMA: 41.35%
• VTLoc ILU iteration 0: ND = 22.97% vs ILU iteration 16: 20.17%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.16146.

Fig 1: An example of a swan in the ObjectFolder Real [7] dataset, demonstrating the effectiveness of our VTLoc method. (a) Tactile contact in the real

Fig 2: The contact point is defined as the projected point of the tactile

Fig 3: VTLoc workflow that predicts coordinates on an object’s surface from tactile readings where the contact occurs. The framework includes a

Fig 4: (b). The tactile data are collected using a Franka Emika

Fig 5 (page 3).

Fig 6 (page 3).

Fig 7 (page 3).

Fig 8 (page 3).

Limitations

• Performance on objects with uniform surface curvature is significantly lower due to intrinsic geometric ambiguities that make tactile contact localization ill-posed.
• Contact localization is inherently ambiguous in cases where different contact points produce similar tactile readings, e.g., flat surfaces or symmetric objects.
• Incorporation of surface normals can degrade performance on uniform objects, indicating sensitivity to noise or overfitting when geometric information is not informative.
• Experiments focus solely on hard rigid objects; extension to soft or deformable objects with variable surface compliance was not addressed.
• Training details such as optimizer parameters, batch sizes, and epoch counts are not fully disclosed, potentially impacting exact reproducibility.
• Though the method generalizes to unknown objects, accuracy drops compared to known objects, suggesting limited robustness to large domain shifts or novel geometries.

Open questions / follow-ons

• How to extend VTLoc to handle soft or deformable objects where surface geometry changes under contact?
• Can the approach be generalized to multi-finger or multi-sensor configurations with more complex tactile feedback?
• How robust is the method under real-world sensor noise, partial or incomplete point clouds, and varying lighting conditions for visual scans?
• Would integrating temporal tactile sequences directly into the model improve localization and disambiguate challenging contacts beyond iterative refinement?

Why it matters for bot defense

From a bot-defense or CAPTCHA perspective, this research demonstrates advanced multi-modal sensor fusion techniques to precisely localize touch contacts on 3D object surfaces, a form of sensory authentication combining vision and touch cues. Although not directly tied to CAPTCHA, the principle of geometric multi-modal alignment and iterative refinement could inspire novel human-verification mechanisms involving spatial reasoning and sensory-matching challenges beyond pure image or interaction patterns. For bot-defense, systems that can reliably correlate tactile sensations with visual geometry introduce a high barrier against spoofing or synthetic manipulation, since mimicking consistent visual-tactile spatial correspondences would be difficult for automated bots without physical presence.

Practitioners designing CAPTCHA challenges or interaction tests involving sensor fusion could consider similar iterative feedback or cross-modal consistency checks to raise the complexity for bots while aligning with natural human perceptual processes. However, the complexity and hardware requirements limit broad deployment, so concepts here would likely be more suitable for highly secure tactile-visual verification contexts rather than mass CAPTCHA use.

Cite

@article{arxiv2607_16146,
  title={ VTLoc: Learning-based Tactile Contact Localization in Visual Point Clouds },
  author={ Zhiyuan Wu and Zhuo Chen and Shan Luo },
  journal={arXiv preprint arXiv:2607.16146},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.16146}
}

Read the full paper

• arXiv:2607.16146 (abstract)
• arXiv:2607.16146 (PDF)