Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs

Source: arXiv:2607.06540 · Published 2026-07-07 · By Zhenyu Liu, Yunxin Li, Xuanyu Zhang, Qixun Teng, Shenyuan Jiang, Haolan Chen et al.

TL;DR

This paper addresses the core challenge of modality interference in native end-to-end full-duplex Spoken Language Models (SLMs), which concurrently process audio input and generate speech output. The authors identify that performance degradation in full-duplex SLMs stems from inherent gradient conflicts between acoustic and semantic modeling when sharing deep parameters, causing substantial knowledge loss and unnatural responses. To overcome this, the authors propose Lychee-FD, a novel hierarchical acoustic-semantic framework that physically separates conflicting modalities in deeper model layers while maintaining cross-modality semantic coherence through a dedicated alignment channel. Extensive experiments on diverse full-duplex benchmarks demonstrate that Lychee-FD significantly advances state-of-the-art speech intelligence (+7.4% accuracy on Spoken QA) and interaction fluidity (+28.5% on FullDuplexBench 1.5) without sacrificing inference efficiency. This work provides the first fine-grained analysis uncovering modality interference’s root cause in full-duplex SLMs and delivers an elegant architectural solution enabling seamless, robust, and intelligent full-duplex spoken language understanding and generation.

Key findings

• Gradient cosine similarity between acoustic and semantic objectives transitions from positive synergy in shallow layers to strong negative conflict (orthogonal/contradictory gradients) in deep layers (Fig 2a), confirming inherent modality interference in shared parameter spaces.
• Semantic gradient magnitudes are suppressed by sparse time-alignment paddings compared to dense text supervision, causing semantic dilution dominated by acoustic gradients (Fig 2b).
• Lychee-FD’s hierarchical parameter separation recovers and exceeds performance of half-duplex backbone StepAudio-2-mini by +0.2% speech-to-text (S→T) and +5.3% speech-to-speech (S→S) accuracy on Spoken QA benchmarks.
• Lychee-FD achieves a 7.4% average absolute accuracy improvement over prior native full-duplex model Fun-Audio-Chat in spoken QA tasks.
• On FullDuplexBench 1.5, Lychee-FD yields +28.5% interaction fluidity improvement vs system-level baselines with external VAD, demonstrating superior turn-taking and interruption management.
• Lychee-FD achieves lowest latency metrics on all benchmarks (First Speech Emit Delay and Interruption Response Delay), validating no inference efficiency tradeoff despite additional architectural complexity.
• Ablation study removing semantic alignment channel causes 5.4%-5.6% performance drop, confirming semantic coherence is essential to knowledge retention.
• Removing hierarchical parameter separation causes catastrophic acoustic performance drop (speech accuracy falls to 27.6%), validating necessity of modality disentanglement.

Threat model

n/a - This paper focuses on architectural and optimization challenges in modeling full-duplex spoken language interactions rather than security or adversarial threat scenarios.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is not explicitly defined as this is a model architectural study aimed at resolving performance challenges in full-duplex SLMs rather than security threats. The assumption is that the model must simultaneously process audio input while generating speech output in real-time, without compromising semantic understanding or acoustic generation.

2. Data: Due to scarce full-duplex datasets, the authors synthesize a training corpus of approximately 140K full-duplex dialogue instances by simulating user-assistant conversations incorporating interruptions, backchannels, and turn-taking dynamics. Speech is synthesized with CosyVoice 2 using 80K voice prompts for diversity. The dataset is rigorously filtered to ensure logical consistency and audio quality.

3. Architecture/Algorithm: The model builds upon a publicly available half-duplex StepAudio-2-mini backbone using the Whisper-v3-large encoder for audio input and CosyVoice2 tokenizer for speech tokenization at 25Hz frame rate. A hierarchical Transformer architecture is proposed consisting of:

   • Shared shallow layers (24 Transformer layers) processing both acoustic and semantic inputs synergistically.
   • Deep-layer parameter separation into three parallel heads: Semantic Head (text), Acoustic Head (speech), and Control Head (interaction management).
   • A Semantic Alignment Channel generating continuous internal monologues during training to anchor semantic learning and counteract gradient dilution caused by sparse time-aligned tokens.

   The total loss is a sum of cross-entropy next-token prediction losses across these modalities.

4. Training Regime: Optimized with AdamW, cosine learning rate scheduler, batch size 32, learning rate 3e-6, warmup ratio 0.1. Training runs for 1 epoch (~16 hours) on 8 NVIDIA H20 GPUs. Three different random seeds are used for averaging results. Inference uses greedy sampling.

5. Evaluation Protocol: Evaluated on three spoken QA benchmarks (LlamaQ, WebQ, TriviaQA) with accuracy in speech-to-text and speech-to-speech modes, and takeover rate measuring conversation turn-taking responsiveness. Full-duplex interaction evaluated on FDBench and FullDuplexBench 1.0/1.5 benchmarks, measuring metrics like Success-Replies Rate, Interrupt Response Latency, Backchannel behavior, and Turn-taking stability. Speech generation quality assessed via Word Error Rate (WER) and UTMOS naturalness scoring. Ablations examine removing hierarchical parameter separation and semantic alignment channel to validate their contributions.

6. Reproducibility: The authors open-source their training pipeline, code, and model weights at the provided GitHub repository. Exact training setup and architecture details are disclosed, enabling replication. However, the synthesized full-duplex datasets are proprietary and not publicly released.

Example End-to-End: Input audio is tokenized and encoded. Shallow shared Transformer layers extract low-level features supporting both text and speech objectives. Deep layers branch into semantic, acoustic, and control heads, producing parallel output token streams. The semantic alignment channel produces an internal textual monologue supervised by continuous text, preserving strong semantic gradient flow. During training, cross-entropy losses for all heads are combined to update the model. At inference, the model simultaneously listens and generates speech tokens in a full-duplex manner, with rapid interruption detection and natural turn-taking driven by the control head.

Technical innovations

• Discovery and formal quantification of gradient conflicts between acoustic and semantic modeling in deep shared layers causing modality interference in full-duplex SLMs.
• Hierarchical parameter separation architecture that retains shared shallow layers but disentangles acoustic, semantic, and control heads in deep layers to resolve optimization divergence.
• Semantic alignment channel that generates internal monologues to maintain high-magnitude semantic gradients and counteract semantic dilution from sparse padded alignment.
• End-to-end native full-duplex framework combining low inference latency with robust knowledge retention, obviating the need for multi-stage training or external VAD modules.

Datasets

• Synthesized Full-Duplex Dialogue Dataset — ~140K instances — Curated via multi-agent simulation with speech synthesized by CosyVoice 2 and pretrained voice prompts (not publicly released)

Baselines vs proposed

• Fun-Audio-Chat (native full-duplex): Spoken QA accuracy (S→S) = 38.8%, Lychee-FD: 42.5%
• StepAudio-2-mini (half-duplex backbone): Spoken QA accuracy (S→S) = 40.9%, Lychee-FD: 46.2%
• VITA 1.5 (system-level pipeline with VAD): Spoken QA accuracy (S→S) = 35.4%, Lychee-FD: 46.2%
• Freeze-Omni (system-level): FullDuplexBench 1.5 average interaction metric improvement +28.5% over baseline, Lychee-FD achieves SIR = 99.7% vs Freeze-Omni 57.2%
• Moshi (native end-to-end): Spoken QA accuracy S→S = 30.5%, Lychee-FD: 46.2%, Speech synthesis UTMOS = 3.72 vs 4.50 for Lychee-FD
• Ablation w/o semantic channel: 5.4-5.6% drop in spoken QA accuracy; w/o hierarchical parameter separation: catastrophic speech accuracy drop to 27.6%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.06540.

Fig 1: Visualization of the efficiency and intelli-

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 6: Layer ablation study on model perfor-

Fig 8: An error analysis illustrating a common limitation in handling side-talk during full-duplex interaction.

Fig 9: Global gradient influence scores among Text and Speech tasks. Left: The fully shared baseline

Fig 7: A case study demonstrating Lychee-FD’s capability in handling complex turn-taking dynamics.

Limitations

• Synthesized training data, while large and diverse, may not capture all real-world full-duplex conversational nuances or noise conditions.
• Semantic alignment channel requires continuous text supervision which may be limited or unavailable in some datasets or languages.
• Ablations show importance of components but no adversarial or robustness evaluation under intentional modality perturbations.
• Evaluation limited to English language datasets and specific architectures; generalization to other languages or larger scale models is untested.
• Full interaction complexity of natural human conversations with overlapping speech and multiple speakers remains a challenge (side-talk handling not fully solved).

Open questions / follow-ons

• Can hierarchical modality separation be dynamically adapted or learned rather than statically assigned to fixed layers?
• How does Lychee-FD perform under noisy or highly overlapping speech scenarios typical in real-world multi-party conversations?
• Is the semantic alignment channel effective if textual transcripts are noisy, partial, or only weakly aligned?
• Can the architectural principles extend to multilingual or code-switched full-duplex spoken language models?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this work presents foundational insights into the intrinsic conflicts arising in multi-modal models that process concurrent speech understanding and generation—key for naturalistic voice-based interaction systems. It highlights the pitfalls of forcing acoustic and semantic objectives into the same deep parameter space, which can degrade semantic coherence and impair model reliability. The hierarchical parameter separation and semantic alignment techniques offer practical approaches to build more robust, responsive voice interfaces capable of seamless interrupt handling and fluid turn-taking without sacrificing inference speed. This is crucial when designing CAPTCHA or bot-detection mechanisms embedded in real-time voice agents, where maintaining semantic integrity and interaction naturalness directly impacts usability and resilience to spoofing or adversarial inputs. Furthermore, the detailed gradient analysis methodology can inspire diagnostics for multi-modal interference in security-critical speech systems.

Cite

@article{arxiv2607_06540,
  title={ Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs },
  author={ Zhenyu Liu and Yunxin Li and Xuanyu Zhang and Qixun Teng and Shenyuan Jiang and Haolan Chen and Minjun Zhao and Fanbo Meng and Yu Xu and Yancheng He and Baotian Hu and Haizhou Li and Min Zhang },
  journal={arXiv preprint arXiv:2607.06540},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.06540}
}

Read the full paper

• arXiv:2607.06540 (abstract)
• arXiv:2607.06540 (PDF)