Toward Real-Time Sentence-Level Sign Language Translation
Source: arXiv:2607.09611 · Published 2026-07-10 · By Thanh-Hoang Nguyen Doan
TL;DR
This paper addresses the challenge of sentence-level sign language translation (SLT) with a focus on real-time deployment, moving beyond isolated sign recognition common in prior work. The authors adapt a frozen SHuBERT encoder pretrained on extensive sign language video with a ByT5 byte-level decoder, fine-tuned via QLoRA on a uniformly sampled 9,872-example subset of the How2Sign dataset. The final model achieves a validation BLEU score of 16.7 and test BLEU of 15.9, with a complementary BLEURT score of 44.7. The main technical contribution is a hardware-aware, streaming system architecture that combines chunked frame ingestion, bounded queues, parallelized MediaPipe-based perception, temporal reordering, and a sentence-boundary finite state machine to enable near real-time sentence-level translation.
Key findings
- Fine-tuning on 9,872-example How2Sign subset achieves validation BLEU 16.7 and test BLEU 15.9, with BLEURT 44.7 on test.
- Concurrent execution of MediaPipe modules reduces per-frame perception latency from 107–111 ms to 45–48 ms.
- Optimized streaming architecture reduces mean post-finalization latency from 1.873 s to 1.354 s (27.71% reduction) over full 9,872-example subset.
- 95th percentile latency improved from 2.919 s to 2.130 s (27.03% reduction) with runtime optimizations.
- Chunked ingestion of frames groups 10 frames sampled at 15 fps at 640×480 JPEG quality 72 with 5 concurrent client requests.
- Sentence-boundary detection based on hand motion and presence states reduces duplicate or premature translation emissions.
- Parameter-efficient QLoRA fine-tuning updates only ByT5 query and value projections and SHuBERT-to-ByT5 projection layer; SHuBERT encoder remains frozen.
- Raspberry Pi 4B prototype runs capture, display, and speech locally while offloading compute-intensive perception and translation to GPU backend with client-agnostic HTTPS interface.
Threat model
The system assumes a trustworthy compute backend and non-adversarial streaming clients; the adversary is not explicitly modeled. The system focuses on real-time accurate translation with limited compute and bandwidth constraints rather than adversarial robustness or attack scenarios.
Methodology — deep read
The threat model here involves a real-time SLT system designed for practical deployment, assuming continuous input video streams from signers captured by a client device, with compute offloaded to a trusted backend. The adversary model is not explicitly adversarial but seeks low-latency, accurate translation under constrained compute and network.
Data is drawn from How2Sign, a large-scale continuous American Sign Language corpus. Due to compute and storage limits, a uniformly sampled working subset of 9,872 video-sentence pairs was extracted. This subset features 9 signers, 7,000 training, 1,121 validation, and 1,751 test examples, with almost all sentences unique, retaining the sentence-level translation task. The videos are processed at 15 fps, resized to 640×480 with JPEG compression.
The architecture is a three-stage cascade: (1) geometric perception uses MediaPipe's face, dual hand, and pose landmark detectors to extract keypoints per frame; (2) regional crops around hands and face are resized and encoded via the DINOv2 self-supervised Vision Transformer into 384-dimensional feature vectors representing pose and expression; (3) SHuBERT, a frozen multi-stream transformer encoder pretrained on 1,000 hours of sign video, fuses the four streams (left hand, right hand, face, upper body pose) at each timestep into a 768-dimensional contextualized sequence representation.
A linear projection maps SHuBERT's output to ByT5 decoder latent space, a byte-level text generator fine-tuned via QLoRA with low-rank adapters and 4-bit quantization on the How2Sign subset. Only the ByT5 query and value attention projections and the SHuBERT->ByT5 projection layer are trainable; all others are frozen to reduce adaptation cost.
Fine-tuning runs on two NVIDIA T4 GPUs with an effective batch size of 512, applying the QLoRA recipe integrating 4-bit NF4 quantization and LoRA rank 4 adapters, training only a fraction of model weights while preserving SHuBERT's pretrained representation.
Deployment separates concerns: a Raspberry Pi 4B client handles video capture, local display, and speech output, sending JPEG-compressed chunks of 10 frames over HTTPS to a CPU/GPU backend for perception and translation. Two concurrent perception workers run MediaPipe modules in parallel via a 4-thread pool, reducing per-frame perception latency.
A sentence-boundary finite state machine governs utterance segmentation based on stable hand presence, motion thresholds, and timing windows, avoiding premature or duplicated sentence finalizations. A reorder buffer ensures temporal ordering of asynchronous perception outputs before translation.
Latency and translation quality are evaluated on the full 9,872-example subset. Latency is measured as the delay between utterance end detected by the state machine and text generation output emitted. Through runtime optimization—including chunk coalescing, concurrent perception, bounded queues, and early loading—the system reduces mean latency by 27.7% and 95th percentile latency by 27.0% compared to a sequential baseline with the same model and decoding parameters.
The Raspberry Pi prototype demonstrates a complete on-device interaction loop, while the system's client-agnostic HTTPS interface supports replacement by browsers, phones, or laptops without backend changes.
Code is publicly released, enabling reproducibility; however, the dataset is partially subsampled due to resource constraints, and hardware-specific latency may vary by client.
Technical innovations
- Use of a frozen SHuBERT multi-stream sign language encoder combined with a byte-level ByT5 decoder fine-tuned via QLoRA for efficient SLT adaptation.
- Hardware-aware streaming pipeline combining chunked ingestion, bounded queues, parallel MediaPipe perception workers, temporal reordering, and a sentence-boundary state machine to reduce translation latency.
- Client-agnostic capture protocol enabling multiple frontends (Raspberry Pi, browser, phone) to use the same backend inference service.
- Concurrent MediaPipe landmark modules executed on separate threads reduce perception latency from over 100 ms/frame to under 50 ms/frame.
Datasets
- How2Sign working subset — 9,872 clip–sentence pairs — uniformly sampled from the original How2Sign corpus
Baselines vs proposed
- Baseline (sequential MediaPipe and single perception worker): mean post-finalization latency = 1.873 s vs optimized streaming system: 1.354 s
- Baseline P95 latency = 2.919 s vs optimized P95 latency = 2.130 s
- Validation BLEU after fine-tuning: 16.7; test BLEU: 15.9; test BLEURT: 44.7
Limitations
- The system translates only after utterance finalization; no truly incremental, word-level streaming translation is implemented.
- Latency results are measured under controlled network and hardware with Raspberry Pi 4B client; results may vary for other clients or under load.
- The How2Sign training and test sets are uniformly subsampled due to compute limits, potentially limiting model generalization and scale.
- The multi-stage pipeline is computationally expensive and the full model does not run entirely on-device, limiting standalone edge deployment.
- Face de-identification via greying and blurring may affect subtle linguistic markers; privacy vs linguistic information tradeoff is not fully quantified.
Open questions / follow-ons
- Can the system be extended to support truly incremental, sub-utterance real-time sign language translation?
- How does network variability and multi-client concurrent use affect real-world latency and throughput?
- What are the impacts of signer diversity, regional dialects, and occlusions on translation quality?
- Can the entire model cascade be efficiently deployed fully on edge devices without backend offloading?
Why it matters for bot defense
For bot-defense and CAPTCHA applications focusing on accessibility or multimodal input, this work shows how multi-stream sign language representations can be translated to natural language sentences in near real-time with hardware-aware engineering. The streaming pipeline design combining chunked input, bounded queues, concurrency, temporal reordering, and finite-state boundary detection offers a potential architectural blueprint for real-time systems requiring low latency and high accuracy in continuous input scenarios. While sign language translation itself is a niche task beyond most CAPTCHAs, the outlined methods for latency reduction, hardware-client abstraction, and parameter-efficient adaptation (QLoRA) may inspire approaches to multimodal authentication or bot-detection involving gesture or motion inputs. The evaluation and design balance accuracy against responsiveness, a key tension in user-facing defensive systems.
Cite
@article{arxiv2607_09611,
title={ Toward Real-Time Sentence-Level Sign Language Translation },
author={ Thanh-Hoang Nguyen Doan },
journal={arXiv preprint arXiv:2607.09611},
year={ 2026 },
url={https://arxiv.org/abs/2607.09611}
}