Semantic Audio-driven Understanding for Dynamic Humanoid Whole Body Control
Source: arXiv:2607.14182 · Published 2026-07-15 · By J. M. A. Marcelo, M. Brienza, E. Bugli, L. Comito, D. Nardi, D. D. Bloisi et al.
TL;DR
This paper addresses the limitation in current humanoid robotic performances that rely heavily on pre-scripted sequences with limited autonomy and responsiveness to dynamic environments. The authors introduce a novel multi-modal orchestration framework that enables a humanoid robot to autonomously select and execute whole-body motion skills in real time driven by continuous audio input. The system distinguishes between music and speech audio streams, using audio fingerprinting and semantic embeddings to align musical segments with corresponding motion policies, while speech input is transcribed and semantically grounded to a discrete skill library for interactive control. Both modalities feed into a unified interface that schedules skill execution over a reinforcement learning-based control pipeline. The approach is validated in simulation using a Unitree G1 humanoid model and further tested on the physical Unitree G1, demonstrating robust sim-to-real transfer and consistent audio-conditioned policy selection in dynamic scenarios.
Key findings
- Overall chunk-level retrieval accuracy of 84.8% on 574 evaluated 5-second audio chunks under temporal offset conditions up to 2 seconds.
- Using a 30-second chunk duration for musical sections yields more stable policy transitions and reduces misalignments compared to 20-second shorter sections (Fig 4 vs Fig 5).
- The system integrates a hierarchical audio routing module to classify incoming chunks into music, speech, or non-relevant categories using Audio Spectrogram Transformer and Silero VAD outputs.
- Fallback to embedding-based retrieval (CLAP model) is employed when audio fingerprint confidence or landmark votes fall below thresholds, maintaining retrieval robustness.
- Speech chunks are transcribed using a low-latency GPT-4o-mini transcription model and mapped via semantic matching to imitation-learned motions, supporting direct human-robot interaction.
- A cool-down mechanism suppresses repeated emissions of identical skill commands within short time windows to damp transition oscillations.
- The control pipeline uses two base policies (stand, walk) and a configurable set of imitation-learned whole-body policies exported as ONNX models for real-time inference.
- Real-world deployment on Unitree G1 confirms stable retrieval-driven policy switching despite higher latencies from external API calls and hardware interface communications.
Threat model
The system assumes an adversary is an environmental audio source that can vary dynamically but is not adversarial or intentionally malicious. The robot must autonomously interpret streaming audio containing music or speech to select appropriate motion policies in real time. It cannot adapt to or defend against corrupted, adversarial, or deeply ambiguous audio signals. The threat model does not consider adversaries attempting to confuse or deceive the semantic retrieval pipeline.
Methodology — deep read
The threat model assumes a dynamic environment where the robot must select appropriate whole-body motion policies from continuous, unstructured audio input streams (music or speech) without explicit pre-scripted timing. No adversarial or malicious audio inputs are considered; the system assumes relatively clean audio to perform semantic grounding.
Data consists of continuous microphone audio segmented into fixed 5-second chunks, processed individually in an online streaming fashion. Evaluation uses mashup scenarios made by concatenating four musically distinct songs that are split into chunks. Temporal offsets of 0.5 to 2 seconds are also tested to validate robustness to misaligned input segments.
The architecture first routes each audio chunk into music, speech, or skip branches. Routing uses two signals: (1) A pre-trained Audio Spectrogram Transformer (AST) classifier produces posteriors aggregated over music and speech classes from the AudioSet ontology, (2) Silero voice activity detector (VAD) generates per-frame speech probabilities. Thresholds on speech fraction and posterior differences decide the branch.
Music chunks are processed by an audio fingerprinting module implementing the Wang constellation-map algorithm indexing track hashes with temporal offsets. Matches are accepted if confidence and aligned landmark votes exceed thresholds. If fingerprinting fails, retrieval falls back to nearest-neighbor search in the CLAP contrastive audio-text embedding space.
Retrieved music matches are temporally grounded to identify the corresponding skill policy to activate, using timed skill rules mapping track segments to imitation-learned controllers.
Speech chunks are first transcribed by a streaming GPT-4o-mini whisper-based model. Transcripts are semantically matched to a discrete skill library to select an appropriate motion. If no match is found, transcripts are forwarded to GPT-4o-mini conversational model to generate spoken responses and trigger time-boxed gestures.
Commands from both modalities feed into a unified TCP socket interface implemented by the RoboJuDo framework, which queues and executes selected skills over the humanoid robot's low-level control loop running either in MuJoCo simulation or on Unitree G1 hardware. Base stand and walk policies mediate stable transitions between imitation-learned motion policies implemented as ONNX models trained via BeyondMimic reinforcement learning framework.
The training of underlying motion policies (BeyondMimic) is outside the scope of this paper but involves imitation learning from human motion capture data exported for real-time control.
Evaluation uses chunk-level retrieval accuracy metric, comparing retrieved skill policy against ground-truth annotations for each audio chunk. Mashup execution scenarios measure temporal alignment of commanded vs executed policy sequences using Gantt charts over 20-second and 30-second interval mashups of distinct songs. Real-world deployment validates sim-to-real transfer fidelity qualitatively.
Code and control software are open source (RoboJuDo), with pretrained models including AST and CLAP publicly available. Exact frozen weights for speech transcription and imitation policies are not detailed, limiting reproducibility of full pipeline. Dataset for controlled mashups is constructed from public songs with manual segmentation and labeled policies.
Technical innovations
- Hierarchical audio routing combining Audio Spectrogram Transformer posteriors with Silero VAD speech fraction to mutually exclusively classify continuous audio chunks into music, speech, or skip categories.
- Temporal grounding of music retrieval using audio fingerprinting with Wang constellation-map algorithm combined with CLAP semantic embeddings fallback to map musical segments to corresponding motion policies.
- Unified multi-modal orchestration framework integrating music- and speech-driven skill retrieval feeding into a single control interface for seamless real-time humanoid whole-body motion policy selection.
- Integration of a cool-down mechanism and confidence thresholds to suppress spurious rapid skill switching at segment boundaries, improving transition stability on physical hardware.
Datasets
- Custom mashup audio chunk dataset — 574 chunks — constructed from Salsa, BTS Dynamite, BTS Swim, Michael Jackson Thriller tracks, publicly available songs
- AudioSet annotator dataset — public audio ontology used to train Audio Spectrogram Transformer classifier
Baselines vs proposed
- Fingerprint-only retrieval accuracy: low when confidence or votes below threshold; fallback to CLAP embedding search improves overall chunk-level accuracy to 84.8% vs audio fingerprinting alone (no explicit metric given)
- Transition stability with 20-second chunks: frequent misalignments and fallback to locomotion policy vs 30-second chunks: consistent timing and stable transitions (Fig 4 vs Fig 5)
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.14182.
Fig 1: Example of human motion retargeting in simulation and real-world de-
Fig 2: Proposed architecture. Continuous microphone audio is segmented
Fig 3: Sim-to-sim execution in the RoboJuDo [34] environment.
Fig 4: Gantt chart for the 20-second mashup (M20). The controller struggles to
Fig 5: Gantt chart for the 30-second mashup (M30). Policy transitions remain
Fig 6: Real-world execution on the Unitree G1, where music recognition drives
Limitations
- Fixed 5-second chunk segmentation may induce latency and misalignment, particularly for short musical segments leading to unstable transitions.
- No explicit adversarial robustness evaluation against noisy or malicious audio input.
- Speech-based grounding limited to discrete skill library, does not handle open-ended natural language behavior generation.
- Real-world evaluation qualitatively described but lacks extensive quantitative performance metrics or user studies.
- Underlying imitation-learned skill training details and datasets are not extensively covered, hindering reproducibility.
- System requires external API calls (speech transcription, LLM) incurring network latency impacting real-time responsiveness.
Open questions / follow-ons
- How can fixed chunk segmentation latency be reduced using adaptive audio segmentation or style-change detection for improved temporal alignment?
- What is the effect of noisy, crowded, or adversarial audio environments on retrieval accuracy and motion selection robustness?
- Can the discrete skill library be expanded or combined with open-ended language models to allow more flexible natural language-driven humanoid control?
- How can transition latency and physical stability be further improved to enable faster and smoother skill switching on real humanoids?
Why it matters for bot defense
While not directly related to CAPTCHA or bot-defense, this work demonstrates a sophisticated semantic retrieval pipeline mapping continuous perceptual input (audio) to discrete behavioral policies in an autonomous system. CAPTCHA engineers could draw inspiration from the hierarchical multimodal routing and confidence-based fallback mechanisms for designing robust real-time classification systems that must handle noisy, overlapping modalities. The temporal grounding approach linking continuous input segments to discrete control outputs offers insight into how asynchronous or misaligned data streams might be managed in behavior orchestration frameworks. Furthermore, the integration of speech transcription with semantic matching illustrates a pathway for natural language interfaces dynamically influencing policy selection, a concept transferable to bot-detection systems that parse active user input signals. However, the technical challenges of latency, transition stability, and error suppression highlighted here are cautionary tale for designing real-time automated classifiers under uncertainty.
Cite
@article{arxiv2607_14182,
title={ Semantic Audio-driven Understanding for Dynamic Humanoid Whole Body Control },
author={ J. M. A. Marcelo and M. Brienza and E. Bugli and L. Comito and D. Nardi and D. D. Bloisi and V. Suriani },
journal={arXiv preprint arXiv:2607.14182},
year={ 2026 },
url={https://arxiv.org/abs/2607.14182}
}