Spoken Function Calling: A New Perspective on Spoken Language Understanding for Large Audio Language Models

Source: arXiv:2608.05126 · Published 2026-08-05 · By Yuezhang Peng, Yuxin Liu, Changfeng Gao, Zhifu Gao, Xiangang Li, Xie Chen

TL;DR

This paper addresses critical limitations in traditional Spoken Language Understanding (SLU) approaches, which rely on closed-set intent classification and slot filling with rigid rules and in-domain fine-tuning, making them brittle for open-domain spoken tasks. The authors propose a new semantic paradigm called Spoken Function Calling (SFC), which reformulates SLU as the task of generating executable structured function calls based on explicitly defined spoken functions with schema constraints. This structure unifies intent and slot parsing, eliminates ambiguity in parameter extraction, and enables better utilization of in-context learning with Large Language Models (LLMs) and Large Audio Language Models (LALMs).

To enable development and evaluation of SFC, they curate 300 spoken functions inspired by traditional SLU datasets, and synthesize a novel large-scale SFC dataset called SFC-Bench using a multi-agent system that generates, validates, and synthesizes multi-turn and multi-intent speech queries with corresponding structured function call labels. They benchmark state-of-the-art LLMs and LALMs on SFC-Bench across multiple difficulty levels, demonstrating that SFC yields 5-18% absolute accuracy gains over traditional SLU on both text and speech inputs. Finally, they enhance LALM SFC capability through RL-based post-training with a fine-grained reward that decomposes function calling correctness into intent, parameter key, and value matching.

Overall, the results show that SFC provides a robust, flexible, and semantically precise representation for spoken semantic parsing that substantially improves model confidence and performance for open-domain audio language understanding tasks, representing a promising new direction beyond closed-set SLU.

Key findings

• SFC improves overall semantic extraction accuracy by 5–18% compared to traditional SLU on both text and speech inputs (Table 3).
• SFC achieves higher function parameter (slot) F1 scores, e.g., Qwen3-8B text input SFC Slot F1: 91.83% vs. SLU Slot F1: 85.20% on Test-ID (Table 3).
• Models output generated under SFC schema have significantly higher generation confidence, with Log Probabilities of -1.62 vs -1.87 for SLU on Test-ID (Table 4).
• LALMs processing speech inputs show substantial accuracy drops due to ASR error propagation, e.g., Qwen3-32B audio accuracy falls from 64.02% (text) to 48.96% (speech) on Test-ID (Table 5).
• Difficulty level strongly impacts performance; top closed-source models see overall SFC accuracy drop from ~80% on Level 1 to ~30% on Level 3-2 (Table 5).
• RL-based post-training with a fine-grained reward that separately scores function name, key matching, and parameter values leads to improved LALM SFC performance.
• Multi-intent and multi-turn SFC tasks remain challenging even for state-of-the-art models, with frequent hallucinations of missing parameters and partial instruction following failures (Table 6).
• SFC-Bench contains 7,000+ samples spanning single-intent, multi-intent, and multi-turn tasks with strict speaker separation for train/test to test generalization.

Threat model

n/a — The paper does not address explicit security adversaries but rather focuses on semantic parsing accuracy in open-domain spoken language understanding. The implicit assumption is benign inputs with linguistic ambiguity and ASR noise, not deliberate adversarial attack.

Methodology — deep read

The paper proposes Spoken Function Calling (SFC), shifting traditional SLU's intent-slot paradigm into directly predicting structured executable function calls with parameters defined via explicit schemas, enabling robust parsing guided by strong formal constraints.

Threat Model & Assumptions: The adversary is not explicitly modeled since this is a semantic parsing task. Instead, the challenge is open-domain task generalization where models cannot rely on memorized intent-slot schemas but must interpret dynamic function definitions. The key assumption is that function schemas are provided as metadata, and environmental context supplements ambiguous or missing speech content.

Data: The authors curated 300 spoken functions across 5 domains and 21 scenarios by transforming existing SLU datasets (like ATIS, SNIPS, SLURP, MAC-SLU) into structured function definitions via an agent pipeline and manual correction. Using a multi-agent system based on GPT-OSS-120B, they generated synthetic spoken queries, corresponding function-call labels, and speech utterances using IndexTTS-2 with LibriSpeech speaker profiles, simulating natural speech phenomena (redundancy, self-corrections). The dataset, SFC-Bench, totals over 7,000 examples with single/multi-intent and single/multi-turn dialogues. Training was on 232 in-distribution functions; testing included 68 out-of-distribution functions to evaluate generalization.

Architecture / Algorithm: For semantic extraction, transformer-based LLMs (e.g., Qwen2.5-7B, Qwen3-8B) and LALMs (Qwen2.5-Omni-7B) were evaluated. Input modalities varied: text queries for LLMs and speech waveforms for LALMs. Output was a sequence of JSON-like function calls. They introduced post-training for LALMs with RL using Group Relative Policy Optimization (GRPO), optimized with a novel fine-grained reward decomposing overall function-call correctness into three parts: function name accuracy, parameter key matching (Jaccard similarity), and parameter value accuracy (normalized edit distance). This encourages learning the components even if full match is initially unattainable.

Training Regime: Supervised fine-tuning (SFT) with Low-Rank Adaptation (LoRA) was first performed for parameter-efficient adaptation. Post-training with RL used sampled output groups evaluated via the fine-grained reward to update the policy. Experiments used vLLM for inference speedup. ASR used Whisper-Large-V3-Turbo with a WER of 13.69% in pipelines.

Evaluation Protocol: The authors compared traditional SLU vs. SFC formulations on identical datasets, evaluated zero-shot and fine-tuned models, and measured separate metrics for function name (intent) accuracy, parameter slot F1, and overall exact-match accuracy. Testing included in-distribution and out-of-distribution splits as well as difficulty levels increasing from single-intent/single-turn to multi-intent/multi-turn with ambiguous parameters. Qualitative error analyses were conducted to identify hallucinations and failure modes. Confidence scores (log probabilities) were also analyzed.

Reproducibility: Code and dataset are publicly available on GitHub, including the SFC-Bench dataset and training/fine-tuning scripts. Although some underlying models are commercial/closed-source, extensive details on dataset curation, agent pipeline, and reward computation are provided to facilitate reproduction. The dataset includes clear train/test splits with speaker isolation.

End-to-end Example: A user query like “Could I add a schedule for a dentist appointment on July 20th at 3 PM in Paris?” is represented in SFC as the function call add_schedule(content='dentist appointment', time='2024-07-20 15:00:00', location='Paris'), rather than separate intent and ambiguous slots. The multi-agent generator creates the spoken utterance, paired text, context metadata, and the labeled function call. The model predicts this structured call directly from speech, maximizing exact match with ground truth function calls under the fine-grained reward during post-training.

Technical innovations

• Introduction of Spoken Function Calling (SFC) as a semantic parsing framework converting spoken queries directly to structured function calls, unifying intent and slot extraction with explicit schema constraints.
• Construction of SFC-Bench, the first large-scale synthetic dataset for multi-turn, multi-intent spoken function calling spanning 300 functions and 7,000+ samples with rigorous train/test speaker isolation.
• Development of a multi-agent generation and validation system employing GPT-OSS-120B and TTS models to synthesize realistic spoken queries, function call labels, and environmental contexts for SFC tasks.
• Proposal of a fine-grained RL reward decomposing function call correctness into function name accuracy, parameter key matching via Jaccard similarity, and parameter value accuracy by normalized edit distance, improving training stability and convergence.

Datasets

• SFC-Bench — 7,078 samples — synthetic dataset constructed from SLU benchmarks using multi-agent system with human validation

Baselines vs proposed

• Qwen3-8B Text Input ICL SLU: Overall accuracy = 69.73% Test-ID vs SFC: 85.70% (+15.97%)
• Qwen2.5-7B Text Input SFT SLU: Overall accuracy = 68.63% Test-ID vs SFC: 83.15% (+14.52%)
• Qwen2.5-Omni-7B Speech Input ICL SLU: Overall accuracy = 53.88% Test-ID vs SFC: 63.86% (+9.98%)
• Qwen2.5-Omni-7B Speech Input SFT SLU: Overall accuracy = 75.61% Test-ID vs SFC: 80.71% (+5.10%)
• Qwen3-32B Text Input SFT Level 1 overall accuracy = 81.71%, drops to 64.63% on speech pipeline
• GPT-4o-Audio Level 1 overall accuracy = 80.27%, drops to 30.31% at Level 3-2
• Gemini-2.5-Pro Level 1 overall accuracy = 78.05%, but only 35.19% at Level 3-2

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.05126.

Fig 1: Comparison of traditional SLU and proposed SFC.

Fig 2: Example of Multi-Intent and Multi-Turn Spoken Function Calling tasks.

Fig 3 (page 1).

Fig 4 (page 1).

Fig 3: Multi-Agent System for Data Construction.

Fig 4: The domain distribution (left) and detailed statistics (right) of average question lengths (tokens), answer lengths

Fig 7 (page 4).

Fig 8 (page 4).

Limitations

• SFC evaluation primarily uses synthetic speech data; real-world spontaneous speech and noise variability remain untested.
• ASR errors severely degrade spoken SFC performance, indicating dependence on upstream transcription quality.
• Current models struggle with multi-intent and multi-turn scenarios with ambiguous or missing parameters, showing hallucinations of wrong values.
• The function schema set, while diverse, is limited to 300 functions in 5 domains; scalability to massively larger function sets is untested.
• No adversarial robustness evaluation against malicious inputs designed to confuse function calls.
• RL post-training fine-grained reward requires precise ground-truth structured annotations, which may be difficult to obtain in other domains.

Open questions / follow-ons

• How well do SFC models generalize to real-world spontaneous speech with background noise and speaker variability beyond the synthetic setup?
• Can the SFC approach be scaled to support thousands of functions and dynamically changing APIs in real-time deployed dialogue agents?
• What defenses or robust training methods can mitigate hallucination of missing or ambiguous parameters in multi-turn/multi-intent scenarios?
• How does the SFC method perform in multilingual or code-switched speech understanding contexts?

Why it matters for bot defense

From a bot-defense and CAPTCHA perspective, this paper offers a compelling advance in speech-based semantic parsing by framing SLU as structured function calling rather than flat intent-slot prediction. This structured representation enables more precise interpretation of spoken commands and can reduce ambiguity exploited by bots attempting to manipulate voice assistants. Implementing SFC-like structured understanding in multimodal audio language models could improve the robustness of voice-based authentication or challenge-response CAPTCHAs by verifying consistent parameter extraction and rejecting ill-formed or hallucinated inputs.

However, the paper also highlights challenges in handling ambiguous or incomplete user queries, especially under noisy automatic speech recognition, which bot adversaries might exploit. Therefore, bot-defense engineers should consider integrating robust multi-turn context modeling and confidence estimation (e.g., via their proposed fine-grained rewards framework) to detect and mitigate fraudulent or ambiguous spoken inputs. Overall, adopting function-calling semantics offers a promising path toward more interpretable, auditable spoken command understanding systems relevant to conversational security challenges.

Cite

@article{arxiv2608_05126,
  title={ Spoken Function Calling: A New Perspective on Spoken Language Understanding for Large Audio Language Models },
  author={ Yuezhang Peng and Yuxin Liu and Changfeng Gao and Zhifu Gao and Xiangang Li and Xie Chen },
  journal={arXiv preprint arXiv:2608.05126},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.05126}
}

Read the full paper

• arXiv:2608.05126 (abstract)
• arXiv:2608.05126 (PDF)