DexCompose: Reusing Dexterous Policies for Multi-Task Manipulation with a Single Hand

Source: arXiv:2606.28323 · Published 2026-06-26 · By Dihong Huang, Zhenyu Wei, Zhuxiu Xu, Yunchao Yao, Sikai Li, Mingyu Ding

TL;DR

DexCompose addresses the challenging problem of composing multiple dexterous manipulation skills with a single robotic hand by explicitly managing finger-level action ownership. Traditional approaches that sequentially chain pretrained manipulation policies suffer from conflicts as the downstream task may overwrite the finger motions needed to preserve the grasp or other initial skill outcomes, leading to destructive interference. DexCompose introduces a role-aware residual framework that reuses pretrained full-hand policies without retraining them end-to-end for every composite task. It first discovers which fingers must be dedicated to preserving the initial skill's outcome via post-task release tests, then composes policies with dual residual modules: a bounded stabilizer that preserves the grasp and a context-aware residual that adapts the downstream policy only within released finger subspaces. On 16 composite dexterous manipulation tasks combining four object-retention and four downstream interaction skills simulated on a Shadow Hand, DexCompose achieves an average composite success of 77.4%, outperforming direct policy chaining by 74.3% and the strongest baseline by 15.8%. The dual-residual stabilizer and explicit finger allocation synergistically reduce interference and stabilize multitask execution. This work outlines a scalable method for flexible dexterous skill reuse through structural action ownership and asymmetric residual correction, advancing beyond standard policy chaining or retraining approaches.

Key findings

• DexCompose achieved 77.4% mean composite success rate on 16 multi-task manipulation benchmarks, surpassing the strongest baseline by 15.8 percentage points (Table 1).
• Direct chained policies had only 3.09% average composite success, demonstrating severe interference between sequential policies without composition.
• Finger-level action ownership and masking improved composite success from 54.5% (ablated -Finger) to 77.4% (full model), highlighting the importance of explicit structural action assignment (Table 2).
• The Task-A residual stabilizer was critical: removing it dropped success from 77.4% to 9.1%, confirming it preserves the initial grasp outcome under disturbances.
• Task-A and Task-B preservation ratios (Fig 3) reached 0.811 and 0.893 respectively with DexCompose, much higher than baselines, indicating effective retention of single-task skills during composition.
• Failure analysis (Fig 4) showed that transition failures dominate and that task-specific interaction dynamics impact robustness, with challenging tasks like TurnOnSwitch causing more object drops.
• The LLM-based finger mask selector outperformed heuristic selection by ~6.5 percentage points on average composite success, demonstrating benefit of task-aware allocation (Table 3).
• DexCompose reused pretrained diffusion-based base policies without retraining them end-to-end, requiring only lightweight residual policy training.

Threat model

n/a — This work focuses on safe and effective composition of dexterous manipulation skills rather than adversarial security threats. The main challenge is endogenous interference between overlapping policies sharing action dimensions, rather than external adversaries attempting to attack or circumvent the system.

Methodology — deep read

1. Threat Model & Assumptions: The setting assumes an adversary is not explicitly modeled; rather, the challenge is internal interference between multiple manipulation policies controlling overlapping action spaces. Policies πA and πB are pretrained full-hand dexterous manipulation policies representing two sequential tasks. Both policies are frozen at composition time. The goal is to compose them so that policy πB can execute downstream tasks without destroying the outcome established by πA (e.g., holding an object). The system assumes access to policy execution in simulation with state observations and actions controlling wrist pose and 22 finger joint positions.

2. Data: Each base task policy is trained on 50 human demonstrations collected in Isaac Lab simulation on the Shadow Hand. For Task A residual stabilizer training, 4096 held grasp states are collected to facilitate training a stabilizing residual policy. The observation space includes proprioceptive joint states and task-specific geometric features; action space is 28-dimensional (3 wrist translation, 3 wrist rotation, 22 finger joint positions). Policies predict 32 future action steps conditioned on last 2 observations.

3. Architecture/Algorithm: Base policies πA and πB are diffusion-based conditional policies trained via behavior cloning with a 1D UNet backbone conditioned by FiLM layers on observation embeddings. DexCompose composes these two frozen base policies using a finger attribution module and dual residual stabilizer.

• Finger Attribution uses held-task post-grasp states and release tests where candidate finger masks release subsets of fingers gradually while measuring object retention (Pret) and clean finger release (Pclean). An LLM-based agent selects an optimal finger ownership mask m* balancing these metrics and residual dexterity.

• Dual Residual Stabilizer composes final action by combining two learned residual modules: πA_res outputs bounded corrections on preserved-finger joints to compensate for disturbances and maintain Task A, while πB_res outputs bounded residuals on downstream policy over wrist and released fingers to adapt to constraints imposed by preserving the grasp.

• Action composition enforces hard ownership masks ensuring Task A controls preserved fingers, Task B controls wrist and released fingers. Residual outputs are tanh-bounded and per-joint clipped.

4. Training Regime: Residual policies πA_res and πB_res are trained with PPO on the Shadow Hand environment with 1024 parallel environments, rollout length 24, for 1000 iterations on a single NVIDIA RTX 4090 GPU. The reward for πA_res penalizes object distance from palm center and residual magnitude to encourage minimal adjustments. Observations include joint states, contact/tactile info, and object pose in hand frame.

5. Evaluation Protocol: Composite success requires both Task A preservation (object retention) and Task B success (task-specific criteria like door opened). For each of 16 composite task pairs (4 Task A × 4 Task B), 256 rollouts across 8 random seeds are evaluated to compute average composite success rates. Baselines include Frozen chaining, Decomposed action space allocation, Residual learning without finger masking, and ablations. Preservation ratios for A and B sides are measured. Failure modes and ablations on each architectural component are analyzed.

6. Reproducibility: Detailed policy architectures, training hyperparameters, simulation setup, and evaluation protocols are provided. Experiments are conducted in Isaac Lab on simulated Shadow Hand. Code and additional details are presumably available (not explicitly stated). However, pretrained base policies use a private dataset of human demos, limiting full replication outside simulation.

Concrete example: For composing GraspBall (Task A) and OpenDoor (Task B) policies, held grasp states after ball grasp are collected. Release tests determine which fingers can be freed without dropping ball, for example releasing ring and little fingers. The Task A residual is trained to compensate for disturbances while holding the ball with preserved fingers. Task B policy acts only on wrist plus released fingers. Final action at each timestep combines these via ownership masks plus learned residuals. Composite success is measured over repeated rollouts balancing retaining grip on ball and successfully opening the door.

Technical innovations

• Post-hoc finger attribution via release tests identifies finger-level action ownership masks from held post-task states, enabling explicit allocation of control between sequential dexterous policies.
• Dual residual stabilizer employs asymmetric residual modules: one bounded residual trained to preserve the initial grasp outcome by stabilizing preserved fingers, another residual to adapt downstream policy only on released fingers and wrist.
• Structural action ownership enforced by action masking within residual learning reduces destructive interference between policies controlling shared action dimensions.
• Use of a learned LLM-based agent for task-aware finger mask selection balances stability and residual dexterity, outperforming heuristic retention-based selection.
• Demonstration that pretrained diffusion-based full-hand policies trained independently on single tasks can be flexibly composed with minimal additional training of lightweight residuals instead of retraining combined policies.

Datasets

• Human demonstration dataset for base tasks — 50 demos per base policy — proprietary, collected in Isaac Lab simulation
• Held post-task states dataset for Task A residual training — 4096 states per Task A skill — simulation-generated

Baselines vs proposed

• Frozen chaining: mean composite success = 3.09% vs DexCompose full: 77.43%
• Decomposed action space: mean composite success = 45.61% vs DexCompose full: 77.43%
• Full residual learning (without finger allocation): mean composite success = 61.60% vs DexCompose full: 77.43%
• Zero-shot DexCompose (no Task-B residual): mean composite success = 69.23% vs DexCompose full: 77.43%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.28323.

Fig 1: DexCompose composes dexterous skills through role-aware finger ownership. By sep-

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• Focuses only on sequential composition of two manipulation skills; extension to longer horizons or multiple interacting skills remains unaddressed.
• Evaluations conducted entirely in simulation; real-world transfer challenges like perception noise or unmodeled contact dynamics are not tested.
• Requires reliable held post-task states and release tests, which may be challenging to acquire outside simulation.
• Pretrained base policies are trained independently on respective tasks; no joint training to explicitly optimize composability.
• Residual policies are lightweight corrections trained with PPO, but hyperparameter sensitivity and stability across tasks could impact results.
• Finger mask assignment relies on an LLM-based agent, which may require task-specific tuning or access to large language models.

Open questions / follow-ons

• How can DexCompose be extended to compose more than two sequential skills or even simultaneous multitask control over a single hand?
• What are the challenges and solutions for transferring DexCompose to real-world robotic hands with noisy sensing and imperfect dynamics models?
• Can finger attribution and residual stabilization be generalized to other robot embodiments or multi-arm systems?
• How does the choice of base policy architecture or training method affect the scalability and success of post-hoc composition?

Why it matters for bot defense

While DexCompose does not directly address bot defense or CAPTCHAs, its core insight—managing and allocating limited high-dimensional actuator resources to compose multiple policies without destructive interference—may inspire analogous strategies in software security contexts. For example, CAPTCHAs or bot-detection systems could benefit from modular policy design where distinct detection or anti-bot interactions are allocated separate 'control' subspaces of user interactions or feature attentions to reduce conflicting signals. Furthermore, the structural ownership and residual stabilizer approach offers a generalizable framework for composing complex decision modules that must coexist without overwriting each other's critical effects. However, practical adaptation to bot defense requires mapping the physical dexterous manipulation setting to cybersecurity interaction spaces, which is nontrivial.

Cite

@article{arxiv2606_28323,
  title={ DexCompose: Reusing Dexterous Policies for Multi-Task Manipulation with a Single Hand },
  author={ Dihong Huang and Zhenyu Wei and Zhuxiu Xu and Yunchao Yao and Sikai Li and Mingyu Ding },
  journal={arXiv preprint arXiv:2606.28323},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.28323}
}

Read the full paper

• arXiv:2606.28323 (abstract)
• arXiv:2606.28323 (PDF)