Casting Everything to Online API Services? A Survey of Integrating Localized Speech Recognition Models in Robotic Systems

Source: arXiv:2607.11792 · Published 2026-07-13 · By Sheng Li, Jing Li, Felix Schijve, Jun Hu, Emilia Barakova

TL;DR

This survey paper addresses the integration of automatic speech recognition (ASR) technologies within robotic systems, a critical enabler of natural and intuitive human-robot interaction. It examines the evolution from traditional hybrid ASR systems to modern deep learning architectures, especially large pretrained speech foundation models like OpenAI's Whisper. The authors comprehensively catalog datasets, toolkits, and deployment strategies (onboard, cloud-based, hybrid) used in robotics. They highlight a variety of real-world robots with speech interfaces, from social humanoid robots (Pepper, Misty II) to personal assistants (Temi, Amazon Astro) and industrial/mobile robots (Boston Dynamics Spot). The survey also discusses key challenges such as noise robustness, latency, speaker diversity, privacy, contextual understanding, and the future promise of multimodal and large-language-model integrated approaches. The paper concludes that while cloud ASR services offer scalable accuracy, onboard solutions protect privacy and responsiveness, making hybrid deployments currently the most practical for robotics.

Key findings

• Large pretrained speech foundation models like OpenAI Whisper (trained on ~680,000 hours of audio) enable state-of-the-art multilingual transcription and are increasingly integrated in robots.
• There are three dominant ASR deployment strategies in robotics: onboard (low latency, high privacy, no network needed), cloud-based (high accuracy, scalable but network and privacy dependent), and hybrid approaches which balance the two.
• ROS-based speech recognition packages exist for several ASR engines, including PocketSphinx, Vosk (based on Kaldi), and Whisper, facilitating onboard processing.
• Popular open datasets include LibriSpeech (~1000 hrs), Mozilla CommonVoice (multilingual, crowdsourced), Switchboard (phone conversations), and larger corpora like Gigaspeech (10,000+ hours) for supervised and semi-supervised learning.
• Robots like Pepper use onboard microphones and ASR for social interaction while devices like Amazon Astro outsource all speech recognition to cloud services (Alexa).
• Onboard ASR deployment faces hardware constraints and noise robustness issues, whereas cloud-based ASR faces latency, connectivity, and privacy concerns.
• Speech recognition alone is insufficient; integration with natural language understanding and multimodal input (vision, gestures) is essential for effective human-robot dialogue.
• Hybrid architectures performing wake-word detection locally and cloud ASR for complex queries currently offer the best compromise between responsiveness and language capability.

Threat model

Not applicable—this is a survey of ASR technology integration in robotic systems rather than a security-focused paper.

Methodology — deep read

As a narrative survey, the authors curated and synthesized representative literature on ASR integration in robotic systems rather than conducting a systematic meta-analysis. They focused the discussion around three dimensions: ASR model families, deployment strategies (onboard, cloud, hybrid), and robotic application domains. The paper reviews established ASR architectures (GMM-HMM, DNN-HMM) and modern end-to-end neural approaches including CTC, attention-based encoder-decoder models (e.g., Transformers), and self-supervised models like Wav2Vec 2.0. They highlight large-scale speech foundation models such as OpenAI Whisper and CMU's OWSM, discussing their architectures and training corpora. The survey catalogs popular datasets (LibriSpeech, CommonVoice, Switchboard, Gigaspeech) used for training these models, detailing dataset sizes and multilingual coverage. The authors describe common ASR toolkits (Kaldi, ESPnet, SpeechBrain) and their applicability to robotics, especially within the ROS middleware framework which supports onboard modular speech recognition nodes. They contrast the deployment trade-offs between onboard processing (requiring computational resources, enabling privacy and low latency) versus cloud ASR services (leveraging powerful but network-dependent APIs like Google Speech-to-Text, Alexa Voice Service). The paper presents examples of robotic platforms using these approaches to illustrate engineering choices, e.g., Misty II uses cloud ASR for complex tasks but offline keyword spotting onboard. They examine challenges such as noise interference, speaker variability, latency, and privacy. While primarily qualitative, the survey suggests careful benchmarking protocols reporting word error rate, latency, hardware footprint, and connectivity assumptions under robotic scenarios for future reproducible study. The paper emphasizes that integration of ASR into robotic natural language understanding and multimodal interaction remains an open challenge. Codebases or weights are not released by this survey because it compiles existing research rather than presenting new empirical models. Overall, the methodology offers a wide-lens view connecting ASR model progress with real-world robotic application constraints in a rapidly evolving research area.

Technical innovations

• Survey highlights the shift from traditional hybrid ASR architectures (GMM-HMM, DNN-HMM) to large-scale pretrained speech foundation models like Whisper that have strong multilingual capabilities.
• Introduces a taxonomy structuring ASR integration in robotics around model family, deployment strategy (onboard, cloud-based, hybrid), and use case domain for systematic analysis.
• Describes emerging hybrid ASR deployment patterns combining local wake-word detection with cloud-based processing to optimize latency, privacy, and accuracy trade-offs.
• Emphasizes the integration of ASR with multimodal inputs (vision, gestures) and large language models toward end-to-end robotic communication systems beyond standalone recognition.

Datasets

• LibriSpeech — ~1000 hours — public audiobook-derived English corpus
• Mozilla CommonVoice — thousands of hours — multilingual crowdsourced corpus
• Switchboard — hundreds of hours — telephone conversational English corpus
• Gigaspeech — 10,000+ hours — large supervised plus semi-supervised English corpus
• Wespeech — 10,000 hours — multi-domain Mandarin dataset
• ReasonSpeech — 10,000+ hours — Japanese speech corpus

Baselines vs proposed

• OpenAI Whisper pretrained model achieves near-human transcription accuracy after training on 680k hours proprietary audio vs traditional GMM-HMM models with far smaller datasets (exact metric deltas not provided).
• Onboard lightweight models like PocketSphinx provide fast, offline command recognition at the cost of lower accuracy vs cloud ASR services like Google Speech-to-Text that require network and incur latency.
• Hybrid approaches retain low latency (onboard wake word detection) with improved accuracy when offloading complex queries to cloud APIs, balancing privacy and responsiveness (quantitative comparisons not standardized).

Limitations

• The survey is a narrative rather than systematic review; lacks reproducible search criteria and may have selection bias toward well-known models and robotic platforms.
• No new empirical evaluation or benchmarking is performed; comparisons are qualitative due to heterogeneous hardware, noise conditions, and task setups across studies.
• Fast-moving ASR and robotics fields mean some described models or deployments may rapidly become outdated.
• Detailed quantitative metrics (e.g., word error rates, latency under standard conditions) are not systematically collected or compared.
• Security and adversarial robustness aspects of ASR in robotics are not deeply explored.
• Integration challenges relating to end-to-end speech-to-action systems remain largely theoretical without concrete empirical validation.

Open questions / follow-ons

• How can robust, low-latency onboard ASR models be designed that approach cloud-level accuracy but remain computationally feasible on embedded robot hardware?
• What are effective methods for integrating ASR outputs with multimodal sensory data and large language models to enable reliable spoken language understanding and safe robot action planning?
• How to best benchmark ASR systems in robotics with unified metrics capturing accuracy, latency, hardware resource usage, and privacy under realistic noisy and dynamic conditions?
• What trade-offs and frameworks can govern privacy-preserving speech recognition in robots operating in diverse environments with multiple users?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this survey illustrates the growing trend of embedding powerful speech recognition models within interactive devices, including robots that rely on voice input as a primary interface. Understanding the deployment trade-offs between onboard and cloud ASR provides insight into latency, privacy, and reliability considerations relevant to real-time automated systems. The documented challenges, such as noise robustness and speaker variability, are analogous to difficulties faced in verifying human presence via voice in security settings. The paper's discussion on multimodal fusion foreshadows future attacks or defenses employing combined audio-visual cues for authentication. Moreover, the surveyed hybrid ASR strategies underscore the importance of layered verification—local simple wake-word detection followed by cloud-based complex processing—which could inspire novel CAPTCHA designs leveraging incremental interaction. However, the paper does not address adversarial attacks on speech recognition or adversarial robustness, so bot-defense engineers should seek complementary research in that area. Overall, this survey equips practitioners to appreciate the practical engineering decisions influencing speech-based human verification in embedded systems.

Cite

@article{arxiv2607_11792,
  title={ Casting Everything to Online API Services? A Survey of Integrating Localized Speech Recognition Models in Robotic Systems },
  author={ Sheng Li and Jing Li and Felix Schijve and Jun Hu and Emilia Barakova },
  journal={arXiv preprint arXiv:2607.11792},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.11792}
}

Read the full paper

• arXiv:2607.11792 (abstract)
• arXiv:2607.11792 (PDF)