Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers
Source: arXiv:2606.31779 · Published 2026-06-30 · By Ying Fan, Anej Svete, Kangwook Lee
TL;DR
This paper addresses the efficiency-accuracy tradeoff in chain-of-thought (CoT) reasoning with large language models (LLMs). Standard explicit CoT generates intermediate reasoning steps autoregressively as tokens, causing latency linear in chain length. Latent CoT methods attempt to compress multi-step reasoning into hidden states to speed inference but have lagged behind explicit CoT in accuracy beyond 1 billion parameters, with the gap widening at larger scale. The authors propose LOTUS, a novel latent reasoning approach using looped Transformers (weight-reused blocks iterated multiple times) to refine a fixed-size latent token prefix in parallel over multiple iterations. They add direct supervision of each latent block via cross-entropy loss on the gold CoT tokens in parallel, bridging the latent-explicit accuracy gap on math reasoning benchmarks at a 3B parameter scale. Experiments on GSM8K and out-of-domain math datasets demonstrate that LOTUS nearly matches explicit CoT accuracy and even surpasses it out-of-domain, while achieving 2.5x to 6.9x speedups in the thought phase. The latent space is shown to be interpretable and aligned with CoT steps. Ablations confirm both the looped architecture and the parallel targeted supervision are key to success. Alternative routing through an auxiliary decoder works but is less robust at smaller scales. This work substantially advances latent CoT methods by combining looped Transformer depth and direct parallel supervision, enabling both faster and more accurate multi-step reasoning at larger model scale.
Key findings
- LOTUS closes the accuracy gap with explicit CoT on GSM8K at 3B parameters, achieving 70.0% vs 71.5% explicit CoT in-domain accuracy.
- On out-of-domain math benchmarks (GSM-Hard, MultiArith, SVAMP), LOTUS surpasses explicit CoT by ~1.5% average at 3B scale.
- Thought-phase latency is reduced 2.5x on math-expression CoT and up to 6.9x on natural-language CoT compared to explicit CoT using LLAMA-3.2-3B backbone.
- The latent supervision loss on gold CoT tokens at each latent block position is essential; removing it drops accuracy by >6 points.
- Looped Transformer iterations (R=6) and latent block width (c=25 tokens) are critical: insufficient loop depth or narrow latent width significantly reduce accuracy.
- Direct supervision through the base LM head is more robust across scales than auxiliary decoder supervision, especially on smaller backbones.
- Readout of post-loop latent states recovers gold CoT intermediate steps and reveals valid alternative reasoning paths, evidencing interpretable latent space alignment.
- Inference-time variations in latent width and loop iterations maintain accuracy within a small margin, demonstrating robustness of the model at test time.
Threat model
n/a — The paper does not define an adversarial threat model but rather focuses on improving latent reasoning architectures for efficiency and accuracy.
Methodology — deep read
Threat model & assumptions: The adversary is implicit as this is a latent reasoning architecture paper without direct security model. The challenge is efficient multi-step reasoning in large LMs without sequential token decoding bottlenecks. Assumes access to gold chain-of-thought reasoning steps for supervised training, no adversarial corruptions.
Data: Uses GSM8K-Aug, which contains 385k math word problems with annotated chain-of-thought rationales. Evaluation on GSM8K test set (in-domain) plus out-of-domain benchmarks GSM-Hard, MultiArith, SVAMP. Also uses a natural-language CoT variant of GSM8K for latency stress tests. Training splits and preprocessing follow prior CoT work; tokenization ensures fixed-length of chain-of-thought steps.
Architecture/algorithm: Builds on causal LLM backbone fθ (GPT-2 124M, LLAMA-3.2-1B, LLAMA-3.2-3B). Introduces K=6 blocks of learnable latent tokens (c=25 tokens per block) forming a padded latent prefix inserted between question and answer. This prefix is refined through R=6 iterations of a looped Transformer architecture that reuses the backbone weights with full backpropagation through the unrolled iterations. The looped Transformer attends jointly to the question key-values cached in a prefix computation.
Training regime: Trained with two complementary cross-entropy losses: (a) Lstep applies parallel supervision at each latent token position to predict the gold chain-of-thought tokens aligned by position, ensuring the latent states represent explicit reasoning steps; (b) Lans applies next-token prediction supervision on the final answer suffix conditioned on the fully refined latent prefix. This constructs a parallel chain likelihood combined with answer selection pressure. Trained on GSM8K-Aug with 3 seeds per setting; hyperparameters fixed across datasets. Optimizer details and seeds in Appendix D (not fully specified here).
Evaluation protocol: Metrics are accuracy on GSM8K and out-of-domain math benchmarks. Latency is measured for query prefill, thought (reasoning), and answer decode phases on an NVIDIA H100 GPU, batch size 1 with greedy decoding. Compares against explicit CoT and prior latent CoT methods including CODI, SIM-CoT, PCCoT, KaVa. Ablations for latent supervision mode, loop iterations R, latent block width c, and inference-time changes in c and R conducted. No explicit cross-validation but use multiple random seeds for statistical confidence.
Reproducibility: Code publicly released at https://github.com/yingfan-bot/lotus. Uses publicly available GSM8K datasets and known model backbones (GPT-2, LLAMA). Exact pretrained checkpoints not indicated for all ablations but seeds and training setups documented. Auxiliary decoder variant LOTUS-aux incorporates an additional autoregressive decoder for CoT supervision during training but inference uses base LM.
Example end-to-end: For a question Q, the model prepares an input sequence forming Q + start-of-thought token + K blocks of learnable latent tokens + end-of-thought token + answer tokens. The question prefix is run once to cache KV states. The looped Transformer iterates R times over the latent prefix and cached KV to refine latent states representing CoT steps simultaneously. After the last iteration, each latent block position is supervised to predict its gold CoT token(s) via cross-entropy, grounding the latent states. Finally, the answer tokens are predicted autoregressively conditioned on these refined latents. At inference, the latent reasoning is done in parallel looped iterations, greatly reducing sequential inference steps compared to explicit CoT token decoding.
Technical innovations
- Use of looped padded Transformers to repeatedly refine a fixed latent prefix in parallel, achieving depth without increasing parameters, enabling compact latent reasoning.
- Parallel cross-entropy supervision directly aligning each latent block token with gold CoT step tokens through the base LM head, grounding latent states explicitly.
- Integration of complementary dual losses: latent step-wise cross-entropy for coverage and answer suffix likelihood for coherent joint reasoning.
- LOTUS-aux auxiliary decoder variant that autoregressively scores CoT tokens conditioned on latent blocks during training, improving supervision routing options.
Datasets
- GSM8K-Aug — 385k training samples — public benchmark with math word problems and CoT rationales
- GSM8K test set — unknown size (standard public split) — public
- GSM-Hard, MultiArith, SVAMP — smaller public math reasoning benchmarks used out-of-domain
Baselines vs proposed
- Explicit CoT (LLAMA-3.2-3B): GSM8K accuracy = 71.5% vs LOTUS: 70.0%
- CODI + SIM-CoT (LLAMA-3.2-3B): GSM8K accuracy = 62.3% vs LOTUS: 70.0%
- KaVa (LLAMA-3.2-3B): GSM8K accuracy gap widens to 5.8 points behind explicit CoT while LOTUS stays within ~1.5 point gap
- Latency (thought phase, LLAMA-3.2-3B): explicit CoT = 338.8 ms vs LOTUS = 133.0 ms (2.5x speedup)
- Latency (thought phase, natural-language CoT stress test): explicit CoT = 963.6 ms vs LOTUS = 140.8 ms (6.9x speedup)
- Ablation no latent supervision: accuracy drops from 70.0% to 63.3% on GSM8K at 3B
- Auxiliary decoder supervision (LOTUS-aux) performs similarly at 3B scale but degrades notably at smaller scales
Limitations
- Evaluations limited to GSM8K and several math reasoning datasets; generalization to other reasoning domains uncertain.
- No adversarial robustness assessment or security-oriented threat evaluation given latent space manipulations.
- Latent block budget K fixed to 6 to cover typical CoT steps; less flexible for problems with more varied step counts.
- Inference-time robustness tested by varying latent width and iterations without retraining but more extreme changes untested.
- Auxiliary decoder routing adds complexity and unstable performance on smaller backbones, limiting practical adoption there.
- Only causal LMs and moderately sized models (up to 3B) studied; scalability to larger models and tasks requires further work.
Open questions / follow-ons
- Can LOTUS’s looped latent reasoning scale effectively to 10B+ parameter models where explicit CoT is extremely costly?
- How generalizable is the latent CoT alignment and interpretability of LOTUS across diverse complex reasoning tasks beyond math?
- Can auxiliary decoder supervision schemes be improved for stable gains across scales and generalized supervision types?
- What are the security implications of interpretable latent states—could they be manipulated or probed adversarially in deployed systems?
Why it matters for bot defense
For bot-defense engineers and CAPTCHA practitioners, the core takeaway is that highly efficient, parallelizable latent reasoning is becoming feasible without accuracy loss, even at moderate model scales. This suggests potential for fast, implicit verification of multi-step reasoning in language models under latency constraints common in bot detection systems. The interpretable latent space of LOTUS also raises interesting possibilities for extracting or verifying intermediate reasoning steps internally, which could be useful for explainability or anomaly detection in automated user behavior analysis. While direct security applications are not explored, the architectural and supervision insights here provide a promising foundation for building latency-sensitive, scalable multi-step reasoning modules that might underpin future bot-defense logic or CAPTCHA challenge validation pipelines.
Cite
@article{arxiv2606_31779,
title={ Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers },
author={ Ying Fan and Anej Svete and Kangwook Lee },
journal={arXiv preprint arXiv:2606.31779},
year={ 2026 },
url={https://arxiv.org/abs/2606.31779}
}