Sensitivity Shaping for Latent Modeling

Source: arXiv:2606.14585 · Published 2026-06-12 · By Hongzhan Yu, Chenghao Li, Ruipeng Zhang, Henrik Christensen, Sicun Gao

TL;DR

This paper addresses a critical failure mode in generative latent dynamics models used for robotic control and planning: local insensitivity to control inputs. When learned dynamics models respond weakly to changes in action inputs near safety-critical regions, predictions under unsupported controls may misleadingly appear similar to training data, causing out-of-distribution (OOD) detection methods based on support surrogates to fail. The authors propose a novel approach called support-conditioned control-sensitivity regularization, which enforces a minimum norm constraint on the control Jacobian of the latent dynamics but only in well-supported regions of the training distribution. Through experiments on multiple robotic tasks including vision-based obstacle avoidance, manipulation, and real-robot navigation, they demonstrate that this regularization enhances the model's sensitivity to control changes, thereby improving OOD detection performance and enabling safer closed-loop planning with fewer undetected failures. Their approach outperforms vanilla dynamics and improves OOD-aware policy filtering across different OOD surrogate methods.

Key findings

• Control sensitivity measured by the Frobenius norm of the latent control Jacobian is low in safety-critical regions with limited training control diversity, causing control-insensitive dynamics.
• Control-insensitive latent dynamics generate predictions under unsupported controls that resemble in-distribution data, suppressing OOD surrogate signals despite large true prediction errors (Fig 1, Fig 4).
• Flow matching OOD surrogates perform best among kNN, ensemble, and flow matching but still miss many safety-critical perturbations without sensitivity regularization.
• Sensitivity-regularized dynamics increase locally measured control-Jacobian norms in near-obstacle regions from values near zero to substantially higher (Fig 6-a).
• Latent transition embeddings separate demonstrated versus undemonstrated controls more distinctly with sensitivity regularization (Fig 6-b,c).
• On a Dubins obstacle avoidance task, sensitivity regularization improves safe trajectory success rates by up to 9-10% and reduces OOD violations by 5-6% across three surrogate types (Table 1).
• In a block-plucking manipulation task, filters trained on sensitivity-regularized dynamics reduce failure rates by ~23-50% compared to baseline filters, with lower reconstruction error and moderate filtering intervention rates (Tables 2-3).
• In a real-robot obstacle avoidance experiment, sensitivity-regularized flow-matching filters achieve 100% success versus 90% for vanilla, with fewer rejections and violations, indicating improved online safety filtering (Table 4).

Threat model

The adversary is an implicit environment or policy shift at deployment time, producing out-of-distribution state-action pairs unseen in training, particularly in safety-critical control regions with limited training control diversity. The adversary can cause unsupported control inputs to the learned dynamics model, but cannot manipulate the underlying training data or force adversarial perturbations beyond natural distribution shifts. The model must detect unsupported transitions relying solely on learned latent support and sensitivity.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is an implicit policy or environment shift causing out-of-distribution (OOD) state-action inputs at deployment time, not modeled during training. The learned latent dynamics model is fixed at test time. The assumption is that safety-critical regions have training data but limited control diversity, causing local control insensitivity. The adversary cannot force arbitrary unseen states but can deviate behavior policies to induce OOD transitions.

2. Data: The primary empirical platform is a vision-based obstacle avoidance system under Dubins dynamics with RGB bird’s-eye images, augmented by robotic manipulation datasets (Franka block-plucking) and real-robot navigation data. Training datasets comprise failure-free trajectories collected via expert and random control policies, producing 200+ collision-free samples for obstacle avoidance. Data splits include held-out calibration sets for conformal OOD thresholds. The data do not include failure or OOD transition states to simulate deployment OOD.

3. Architecture / Algorithm: The model builds on DreamerV3's recurrent state-space model (RSSM) latent dynamics, which encodes observations into latent states z_t and models transitions p(z_{t+1}|z_t,u_t). The novel component is a support-conditioned sensitivity regularizer L_reg = E_{(z_t,u_t)∈ D_bar} [max(0, g - ||∂F_θ(z_t,u_t)/∂u_t||_F)^2], which enforces a minimum Frobenius norm g on the latent dynamics' control Jacobian but only on a high-support subset of training data identified by low kNN distances. The encoder and dynamics are jointly trained with the DreamerV3 variational objective plus this Jacobian sensitivity penalty weighted by λ_reg.

4. Training Regime: The model trains for an unspecified number of epochs (details deferred to appendix), periodically updating the high-support set by selecting β fraction of samples with smallest local latent kNN distances. The Jacobian norm is approximated using Hutchinson’s estimator for efficiency. Hyperparameters g, β, and λ_reg are tuned to balance sensitivity enforcement without incurring large extrapolation errors outside support.

5. Evaluation Protocol: OOD surrogates evaluated include kNN latent transition distances, ensemble model disagreement, and flow matching density estimates, each converted to calibrated OOD thresholds via inductive conformal prediction at coverage levels (e.g., 0.90). Evaluation metrics include success rates (ρ_succ), failure rates (ρ_fail), filtering rejection rates (ρ_filt), and OOD violation rates (ρ_viol). Ablations contrast vanilla vs regularized dynamics on OOD detection sensitivity and downstream safety filtering. Real-robot tests validate transfer.

6. Reproducibility: Code and trained models are not explicitly released. Datasets include simulated Dubins and proprietary robot data, limiting external reproduction. Architectural and hyperparameter details are partly deferred to supplemental material. The paper undertakes comprehensive ablation through multiple OOD surrogates and datasets.

Example Pipeline: For the Dubins obstacle avoidance, the model is trained on collision-free images and controls, then latent control-Jacobian norms are computed and encouraged to exceed a threshold g in high-support regions defined by kNN on latent transitions. At test time, the deployed policy's candidate controls are filtered by computing surrogate OOD scores (e.g., kNN distance). Controls exceeding calibrated thresholds are rejected in favor of safer alternatives. The regularized dynamics produce more sensitive predictions, causing OOD surrogates to better flag unsupported controls before failures occur, raising safe trajectory success rates by 20% (Table 1).

Technical innovations

• Identification of control insensitivity in latent generative dynamics models as a fundamental cause of OOD detection failures.
• Support-conditioned sensitivity regularizer that enforces non-vanishing control-Jacobian norms only in well-supported latent regions, avoiding unstable extrapolation in weakly supported areas.
• Use of latent kNN-based support estimation combined with Jacobian norm regularization to selectively shape local dynamics sensitivity.
• Demonstration that control sensitivity shapes the learned latent transition geometry to better separate in-distribution and OOD transitions, enabling safer control filtering.

Datasets

• Vision-based Dubins obstacle avoidance — ~200 failure-free trajectories — simulation/controlled collection
• Franka block-plucking manipulation — 100+ trials — lab-collected wrist and tabletop RGB + proprioception
• Real-robot obstacle avoidance — 60 minutes teleoperated data — dual body-camera images with discrete control commands

Baselines vs proposed

• Vanilla dynamics + kNN OOD surrogate: success rate 0.60 vs proposed regularized dynamics + kNN: 0.80 (Dubins obstacle avoidance, Table 1)
• Vanilla dynamics + flow matching: 0.91 success vs proposed: 0.96 (Table 1)
• Vanilla dynamics + ensemble: 0.38 success vs proposed: 0.44 (Table 1)
• Block-plucking median policy + vanilla ensemble OOD filter: failure rate 0.196 vs proposed: 0.150 (Table 2)
• Block-plucking imagined rollout policy + proposed filter (α=0.1): failure rate 0.047 vs vanilla filter 0.118 (Table 3)
• Real-robot obstacle avoidance + vanilla flow matching filter: success 0.90 vs proposed: 1.00 (Table 4)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.14585.

Fig 1: Overview of sensitivity shaping for reliable OOD detection in generative dynamics models. Control-

Fig 2: Dubins system. (Left) RGB obesrvations. Over-

Fig 3: Sensitivity analysis in the obstacle-avoidance system. From left to right: (a,b) Control-Jacobian

Fig 4: OOD surrogate responses to perturbations. (Left) Average surrogate scores over near-obstacle sam-

Fig 5: Flow-matching scores for the state

Fig 6: Sensitivity analysis with sensitivity-regularized dynamics. From left to right: (a) Control-Jacobian

Fig 7: Observations

Fig 8: Real-robot obstacle avoidance

Limitations

• Relies on quality of latent space kNN to identify high-support regions; poor representations or distance metrics may degrade support estimation accuracy.
• Support-set updates introduce computational overhead, which may not scale with very large datasets or online training.
• Does not guarantee accurate dynamics predictions outside supported regions, only improved local sensitivity where data is dense.
• Limited adversarial or worst-case distribution shift evaluation; focuses on typical OOD induced by policy deviations.
• Experiments focus on vision-based robotics with relatively low-dimensional controls; generalization to high-dimensional or discrete control spaces requires further validation.

Open questions / follow-ons

• How to scale support-conditioned sensitivity regularization to very large or continually changing datasets with online updates?
• Can alternative latent distance metrics or learned support estimators improve identification of high-support regions, reducing false negatives in OOD detection?
• How does sensitivity shaping interact with adversarial robustness or worst-case OOD scenarios beyond policy-induced shifts?
• What are the impacts of sensitivity regularization on very high-dimensional or discrete action spaces common in robotics?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this paper highlights a fundamental challenge in learned latent dynamics models related to control sensitivity. When models respond weakly to certain inputs, OOD detectors relying on support estimation risk being blinded to policy-induced anomalies, analogous to bot behaviors that produce seemingly in-distribution patterns despite divergent intentions. The proposed support-conditioned sensitivity regularization suggests a way to enforce local input sensitivity during model training, improving the detectability of unsupported or adversarial actions. This insight cautions against solely relying on post hoc OOD surrogate measures in systems governed by learned latent representations and encourages integrating sensitivity conditioning within model training—a strategy that may translate to latent behavioral models in bot detection or CAPTCHA interaction analytics. Practitioners should consider how local input sensitivity shapes the geometry of latent states and the ensuing efficacy of support-based anomaly detection.

Cite

@article{arxiv2606_14585,
  title={ Sensitivity Shaping for Latent Modeling },
  author={ Hongzhan Yu and Chenghao Li and Ruipeng Zhang and Henrik Christensen and Sicun Gao },
  journal={arXiv preprint arXiv:2606.14585},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.14585}
}

Read the full paper

• arXiv:2606.14585 (abstract)
• arXiv:2606.14585 (PDF)