Dziri Voicebot: An End-to-End Low-Resource Speech-to-Speech Conversational System for Algerian Dialect

Source: arXiv:2606.26003 · Published 2026-06-24 · By Dihia Lanasri, Fairouz Taki, Asma Kemmoum

TL;DR

This paper tackles the challenge of building a complete end-to-end speech-to-speech conversational system for Algerian Dialect (Darija), a severely low-resource, code-switching, and linguistically complex dialect widely spoken by over 45 million people. The authors propose a modular architecture that integrates adapted automatic speech recognition (ASR), natural language understanding (NLU), retrieval-augmented generation (RAG), and text-to-speech synthesis (TTS) components tailored to Algerian Dialect in a telecom domain. They construct and curate new datasets for ASR (2.68 hours, 4,103 utterances), NLU (15,891 labeled examples), and TTS (50.7 minutes single speaker), and fine-tune large pretrained multilingual models such as Whisper for ASR, DziriBERT for NLU, and XTTS-v2 and VITS-ar for TTS. Experimental results show the Whisper-based ASR achieved a 13.74% word error rate (WER), improving by over 50% relative to Wav2Vec2 baselines. The NLU system achieves 98.4% intent classification accuracy and 93.9% entity F1, while the RAG module enables handling open-domain queries with a composite score of 78.5/100. The first fine-tuned TTS models for Algerian Dialect produce intelligible natural speech for telecom scenarios. The end-to-end system represents the first integrated speech-to-speech voicebot for Algerian Dialect with publicly documented training and evaluation, establishing a reproducible baseline for future research.

Key findings

• Whisper-medium ASR model fine-tuned on 2.68h telecom Algerian Dialect corpus achieves 13.74% WER, outperforming Wav2Vec2-XLS-R by ~53% relative (from 29.04% to 13.74%) on the test set of 411 utterances.
• Inclusion of out-of-domain datasets (e.g., CASABLANCA corpus) during ASR training degrades performance significantly, highlighting domain mismatch sensitivity in low-resource ASR.
• NLU component combining DziriBERT embeddings with Rasa framework achieves 98.4% intent classification accuracy and 93.9% entity-level F1-score on a telecom domain dataset of 15,891 examples spanning 80 intents and 28 entities.
• Retrieval-Augmented Generation fallback module using FAISS semantic search plus locally deployed Llama 3.2 3B model obtains a composite score of 78.5/100 on open-domain queries while maintaining factual consistency.
• The telecom-specific Algerian Dialect TTS corpus of 50.7 minutes enabled successful adaptation of two architectures: XTTS-v2 (with LoRA) and VITS-ar, producing stable and intelligible synthetic speech.
• Manual ASR error analysis shows Whisper errors are mostly phonetic and preserve meaning, enabling downstream NLU to remain effective, whereas Wav2Vec2 incurs more semantic errors affecting intent detection.
• Natural speaker and linguistic variability introduced via 14 speakers with diverse gender, prosody, and code-switching improves model robustness despite limited dataset size.
• The end-to-end integrated pipeline enables spoken input and output interaction, the first for Algerian Dialect beyond prior text-only dialogue systems.

Threat model

The paper does not explicitly define a security threat model; the main adversary is implicitly the natural variability and complexity of spontaneous Algerian Dialect speech with code-switching and out-of-domain queries. No adversarial attacks or malicious users are considered. The focus is on realistic voicebot interactions in a telecom domain, assuming non-adversarial users speaking diverse dialectal variants. The system does not address robustness to deliberate evasion, poisoning, or spoofing attacks.

Methodology — deep read

The authors address the problem of developing a complete end-to-end conversational voicebot pipeline for Algerian Dialect, a low-resource and code-switching dialect lacking standard orthography and large corpora. The threat model centers around realistic user interactions primarily in the telecom domain, with input speech exhibiting lexical, phonetic variability and French-Arabic code-switching. The adversary in a security sense is not directly addressed.

Data was collected specifically for this task: the ASR corpus consists of 4,103 utterances (approx. 2.68 hours) recorded from 14 Algerian speakers in the telecom domain, covering 70 distinct intents with natural code-switching present. The dataset was split 80/10/10 into train, validation, and test sets with random seed 42 for reproducibility. Audio was downsampled to 16 kHz mono and transcriptions normalized for orthographic variation.

For the ASR component, two pretrained multilingual speech models (Whisper-medium and Wav2Vec2-XLS-R-300M) were fine-tuned. Both employed the same train/val/test splits and preprocessing. Whisper models were trained using two stages with decreasing learning rates focusing on acoustic adaptation and polishing; Wav2Vec2 fine-tuning used CTC loss with data augmentation (noise, volume). Out-of-domain datasets (CAFE, CASABLANCA) were selectively included to study the impact on domain adaptation. Model selection was based on minimum word error rate (WER).

NLU and Dialogue Management employed the Rasa framework with DziriBERT transformer-based embeddings for intent classification and entity extraction on a large in-domain labeled dataset (15,891 utterances spanning 80 intents and 28 entities). Dialogue management combined rule-based and learned policies for flexibility and consistency. A Retrieval-Augmented Generation (RAG) fallback was introduced with semantic retrieval from a telecom knowledge base indexed by FAISS and generation via a locally deployed LLaMA 3.2 3B instruct model constrained by prompt engineering to minimize hallucinations.

TTS models were trained on a separately collected 50.7-minute single-speaker Algerian Dialect telecom corpus recorded at 44.1 kHz, split 95/5 for train/eval. Fine-tuning strategies included parameter-efficient LoRA adaptation of XTTS-v2 (a GPT-based acoustic model with HiFi-GAN vocoder) and partial decoder-only fine-tuning of VITS-ar, a VITS derivative for Arabic. Both models aimed to capture dialectal phonetic nuances and code-switching.

Evaluation protocols included WER on a held-out ASR test set (411 utterances), intent classification accuracy and entity F1 for NLU assessed in prior work, and composite metrics for RAG module applied on open-domain queries (details not fully provided). TTS quality was measured subjectively and via standard loss metrics, though exact test procedures for TTS were less detailed. Cross-validation was not mentioned; random splits with fixed seeds were used for reproducibility.

The system was deployed as a modular pipeline where ASR outputs feed NLU, passed to dialogue management and RAG, culminating in TTS generation of responses. This design allows independent evolution of modules with seamless integration. The code base and datasets are mentioned as curated and structured for reproducibility, but public release status is unclear. Specific hardware or training duration details are not specified. Overall, the pipeline combines transfer learning from large pretrained models with domain- and dialect-specific adaptations under realistic low-resource constraints, enabling the first integrated Algerian Dialect speech-to-speech conversational system.

A concrete example: A user speaks a telecom-related utterance in Algerian Darija with French code switches. The Whisper-medium ASR converts speech to text with a 13.74% WER. The text passes to DziriBERT-enhanced Rasa NLU, which detects the intent and extracts entities accurately. If the intent matches a predefined action, the Dialogue Manager generates a response; otherwise, a RAG component retrieves related knowledge and generates a reply text through the LLaMA 3.2 model. Finally, this textual response is rendered into natural speech via XTTS-v2 (LoRA) or VITS-ar TTS models fine-tuned on telecom Algerian Dialect recordings.

Technical innovations

• Fine-tuning Whisper-medium ASR on a small 2.68h Algerian Dialect telecom dataset with signal-based augmentation to achieve a 13.74% WER, outperforming Wav2Vec2 baselines.
• Integration of a hybrid rule-based and learned dialogue management combined with DziriBERT-based NLU and Rasa framework adapted to dialectal Arabic in a speech pipeline.
• Conditional Retrieval-Augmented Generation fallback combining FAISS semantic retrieval of telecom knowledge bases with a locally deployed Llama 3.2 3B model constrained by prompt engineering for factual consistency.
• Application of parameter-efficient Low-Rank Adaptation (LoRA) for XTTS-v2 and partial decoder-only fine-tuning for VITS-ar TTS architectures to enable low-resource Algerian Dialect speech synthesis.

Datasets

• Algerian Dialect Telecom ASR Corpus — 2.68 hours (4,103 utterances, 14 speakers, 70 intents) — proprietary collected for this study
• Algerian Dialect Telecom NLU Dataset — 15,891 examples, 80 intents, 28 entities — proprietary
• Algerian Dialect Telecom TTS Corpus — 50.7 minutes single speaker — proprietary

Baselines vs proposed

• Wav2Vec2-XLS-R (telecom only): WER = 31.67% vs Whisper-medium (telecom + signal augmentation): WER = 13.74%
• Wav2Vec2-XLS-R (telecom + CAFE corpus): WER = 29.04% vs Whisper-medium (telecom + Casablanca corpus): WER = 36.46% (performance degradation)
• DziriBERT + Rasa NLU on telecom dataset: intent accuracy = 98.4%, entity F1 = 93.9%
• RAG module with Llama 3.2 3B: composite performance score 78.5/100

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.26003.

Fig 1: Overview of the proposed DziriVoice end-to-end architecture.

Limitations

• Relatively small ASR training corpus (2.68 hours) limits potential generalization beyond telecom domain and dialect varieties.
• TTS corpus is single speaker and limited in duration (50.7 minutes), restricting speaker variability and naturalness evaluation.
• No reported adversarial evaluation or robustness testing against spontaneous speech, noisy environments, or uncurated user utterances.
• NLU and RAG components evaluated primarily in prior work with textual input; impact of noisy ASR transcripts on end-to-end spoken system is not extensively measured.
• Integration evaluations focus mainly on component-level metrics rather than full dialogue system interaction quality or user experience studies.
• Deployment details, computation costs, and latency of the pipeline are not discussed, limiting practical assessment.

Open questions / follow-ons

• How well does the end-to-end system perform in fully interactive, multi-turn spoken dialogue scenarios under real-world acoustic and conversational variability?
• What is the impact of ASR errors on downstream NLU, dialogue management, and response generation in end-to-end interaction, including error propagation effects?
• How effective are the TTS synthesis methods in conveying naturalness and speaker characteristics across multiple speakers or expressive styles in Algerian Dialect?
• Can unsupervised or semi-supervised techniques further alleviate data scarcity for Algerian Dialect ASR, NLU, and TTS components?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this paper highlights the complexity and challenges in building conversational voice systems for low-resource, code-switching dialects like Algerian Darija. The demonstrated pipeline integrating ASR, NLU, retrieval, and TTS underlines the importance of domain- and dialect-specific adaptations when designing voice-based authentication or anti-bot measures. The relatively high ASR error rate (13.74% WER) even with fine-tuning indicates that voice-based authentication systems must be robust to recognition noise and dialectal variation. The successful use of retrieval-augmented generation to handle open-domain queries suggests potential strategies to validate user intent or behavior through semantic analysis. However, limitations such as small corpora and limited adversarial evaluation caution against direct deployment for security-critical applications without further robustness testing. Overall, the work informs how low-resource speech systems might be integrated or protected within broader anti-bot frameworks requiring spoken human verification.

Cite

@article{arxiv2606_26003,
  title={ Dziri Voicebot: An End-to-End Low-Resource Speech-to-Speech Conversational System for Algerian Dialect },
  author={ Dihia Lanasri and Fairouz Taki and Asma Kemmoum },
  journal={arXiv preprint arXiv:2606.26003},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.26003}
}

Read the full paper

• arXiv:2606.26003 (abstract)
• arXiv:2606.26003 (PDF)