A Practical Investigation of Training-free Relaxed Speculative Decoding
Source: arXiv:2607.08690 · Published 2026-07-09 · By Guoxuan Xia, Luka Ribar, Paul Balanca
TL;DR
This paper investigates training-free relaxed speculative decoding methods to accelerate the sampling process in autoregressive large language models (LLMs). Speculative decoding typically uses a fast auxiliary drafter model to generate token drafts, which are then verified in parallel by the main LLM verifier model, ensuring strict distribution preservation and thus lossless acceleration. Recent work proposes relaxing this distribution preservation to allow faster decoding, trading off some fidelity to the original verifier distribution for speed or even capability gains. The authors present a unified framework to describe and compare various relaxed speculative decoding techniques, benchmark them across contemporary drafter-verifier pairs and reasoning tasks, and analyze practical speed-accuracy trade-offs. They find that relaxed approaches require more extensive capability evaluation compared to strict speculative decoding and often depend on drafted tokens coming from well-aligned language models, limiting the suitability of lightweight dedicated draft modules.
Key findings
- Relaxed speculative decoding techniques can achieve higher speed-ups than strict speculative decoding by accepting more draft tokens, but require capability re-evaluation due to deviating from the verifier distribution.
- Strict speculative decoding preserves the original LLM sampling distribution via rejection/resampling, eliminating the need for costly capability re-evaluations.
- Speed-up is primarily governed by the drafter's relative cost to the verifier (crel) and the average number of accepted draft tokens (¯laccept), with optimized draft length (Ndraft) interacting with both.
- Reducible fuzzy speculative decoding (r-fuzzy) has provably better or equal expected speed-up compared to fuzzy speculative decoding while sampling from the same relaxed distribution (Appendix C).
- Relaxed speculative decoding approaches generally rely on draft distributions q that are well-aligned with verifier p, limiting the utility of lightweight multi-token-predictors (MTP) as drafters.
- Empirical results highlight that, for some relaxed methods (e.g., spec-casc-opt, CACTUS), capability can improve slightly, possibly due to ensembling effects.
- Using a proxy speed-up model S ≈ (¯laccept + 1)/(1 + Ndraft * crel) enables reasoning over trade-offs across hardware/software setups, overcoming variability seen in real-world speed measures.
- Relaxation parameters α allow tuning capability-speed trade-offs, but tuning is expensive for frontier models with broad capabilities.
Threat model
N/A — the paper focuses on decoding acceleration methods and trade-offs rather than on adversarial threats or security considerations.
Methodology — deep read
The authors start by formalizing strict speculative decoding with a drafter model q and main verifier model p, where draft tokens are stochastically accepted or rejected based on the ratio p(x)/q(x), guaranteeing sampling from p exactly (Algorithm 1).
They develop a generalized framework for relaxed speculative decoding (Algorithm 2) that replaces the exact verifier target distributions with relaxed targets πrej, πres, and πbonus parameterized by a relaxation α. This allows accepting more draft tokens at the cost of deviating from p, trading accuracy for speed.
They collect diverse recent relaxed decoding methods under this framework, from CACTUS (reverse KL constrained optimization), mentored decoding (forward KL constrained), r-fuzzy (divergence-based acceptance), spec-casc-opt (confidence-based model cascading), ensemble weighted distributions, to spec-contrastive decoding which modifies logits for improved capability.
For benchmarking, they consider multiple drafter-verifier pairs representing practical LLM inference settings including large models (e.g., 32B verifier with 0.6B drafter). They measure draft token acceptance rates (¯laccept) at several draft lengths (Ndraft) and estimate speed-up using a proxy model involving relative drafter cost crel derived from hardware latency measurements.
They evaluate capability trade-offs through representative reasoning benchmarks, comparing strict and relaxed decoding accuracy across relaxation parameters. They highlight the need to consider changes in average generation length (¯L) when computing speed-up for relaxed decoding.
The paper analyzes implications on efficiency when integrating lightweight multi-token prediction drafters, showing many relaxed methods depend on strong drafter capability which lightweight dedicated MTP modules lack.
They provide a taxonomy (Table 2) summarizing each method’s rejection and sampling distributions, narratives, and relaxation strategies for interpretation and comparisons.
Code and scripts for reproducing experiments, including measurement of latencies on NVIDIA GH200 GPUs using vLLM, are released enabling further validation.
One concrete example: using a Qwen3 0.6B drafter with a 32B verifier, they measure Cdrafter and CAR timings on GPU and evaluate strict and relaxed decoding acceptance rates and accuracy on reasoning tasks at varying Ndraft and relaxation α, illustrating real speed/runtime trade-offs in practice.
Limitations in prior real speed measurements on vLLM due to high drafter relative cost crel motivate use of their speed-up proxy model for reasoning across deployment scenarios. The paper provides practical guidance on choosing Ndraft and α for real-world setups.
Overall, the methodology is a thorough empirical and theoretical comparison of training-free relaxed speculative decoding methods unified under a common abstraction, evaluated on modern hardware and tasks with attention to real deployment considerations.
Technical innovations
- A unified framework (Algorithm 2) that represents diverse relaxed speculative decoding methods using parameterized relaxed target distributions πrej, πres, and πbonus.
- A proxy speed-up model S ≈ (¯laccept + 1) / (1 + Ndraft * crel) that captures interplay between drafter cost, acceptance length, and draft size to guide practical deployment.
- Formal proofs that reducible fuzzy speculative decoding improves speed-up while sampling from the same relaxed distribution as prior fuzzy spec-dec, clarifying theoretical benefits.
- Construction of a taxonomy (Table 2) that categorizes recent relaxed speculative decoding methods by their relaxation strategies, divergence constraints, and targeting narratives.
- Empirical identification that relaxed speculative decoding methods typically depend on well-aligned language model drafters, limiting applicability of lightweight dedicated multi-token-prediction modules.
Baselines vs proposed
- Strict speculative decoding: preserves verifier distribution and requires no capability re-evaluation; speed-up depends on acceptance rates and drafter cost.
- CACTUS relaxed spec-dec (Hao & Mou 2026): achieves close-to-lossless speed-up with controlled KL divergence, sometimes improves capability vs strict spec-dec.
- Mentored decoding (Tran-Thien 2023): similar capability-speed trade-off behavior as CACTUS under forward KL relaxation.
- Reducible fuzzy speculative decoding (Holsman et al., 2025): provably higher speed-up and same relaxed distribution compared to fuzzy speculative decoding.
- Spec-casc-opt (Narasimhan et al., 2025): uses confidence-based deferral with speed-ups close to lossless speculative decoding and possible capability gains.
- Ensemble method (Wang et al., 2026): weighted avearge of drafter and verifier distributions yields improved acceptance rates and capability.
- Spec-cont-dec (Yuan et al., 2024): uses contrastive logits to improve capability with relaxed speculative decoding but requires use of relaxed πbonus.
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.08690.
Fig 1: Left: In strict speculative decoding tokens are rapidly drafted and then verified in parallel. Draft tokens are
Limitations
- Relaxed speculative decoding requires expensive capability re-evaluation for tuning relaxation parameter α, limiting ease of deployment on generalist LLMs.
- Speed-up gains depend heavily on the quality and alignment of the drafter model q; lightweight low-cost multi-token predictors often do not satisfy this requirement.
- Real-world speed-up measurements are sensitive to hardware/software stacks and drafter-verifier relative costs (crel), complicating reproducibility across platforms.
- While relaxed methods can sometimes improve capability, results are inconsistent and not guaranteed beyond strict spec-dec.
- The evaluation focuses on training-free methods; learned relaxed decoding approaches with auxiliary training are out of this study's scope.
- Distribution deviations introduced by relaxation affect response length and quality metrics, complicating fair evaluation across setups.
Open questions / follow-ons
- How do training-based relaxed speculative decoding methods compare in efficiency and capability trade-offs versus training-free approaches?
- Can the relaxed speculative decoding framework be extended to leverage more diverse auxiliary models such as multimodal or retrieval-augmented drafters effectively?
- What are principled methods to automate tuning of relaxation parameter α to balance capability and speed under diverse deployment constraints?
- How do relaxed speculative decoding methods behave under distribution shifts or adversarial inputs that cause drafter-verifier mismatch?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, this paper provides a thorough empirical and conceptual understanding of how relaxed speculative decoding can accelerate LLM inference with configurable trade-offs between generation quality and latency. While CAPTCHAs involve generation tasks with stringent correctness and security criteria, knowing that relaxed techniques may compromise distribution fidelity explains the risks of reduced fidelity or unpredictable outputs. Practitioners deploying multi-token prediction modules for CAPTCHA text generation should be cautious: relaxed speculative decoding typically requires well-aligned drafter models and incurs capability evaluation overhead to safely tune. The presented proxy speed-up model offers a practical tool to gauge latency improvements under various hardware and model cost assumptions, useful for latency-sensitive interactive security applications. However, the dependency of relaxed methods on the drafter quality and the need for costly capability re-evaluation may limit adoption in security-critical CAPTCHAs requiring strict output guarantees.
Cite
@article{arxiv2607_08690,
title={ A Practical Investigation of Training-free Relaxed Speculative Decoding },
author={ Guoxuan Xia and Luka Ribar and Paul Balanca },
journal={arXiv preprint arXiv:2607.08690},
year={ 2026 },
url={https://arxiv.org/abs/2607.08690}
}