In-Place Tokenizer Expansion for Pre-trained LLMs

Source: arXiv:2607.15232 · Published 2026-07-16 · By Jimmy T. H. Smith, Tarek Dakhran, Alberto Cabrera, Simon S. Lee, Paul Pak, Aditya Tadimeti et al.

TL;DR

This paper addresses the problem of fixed tokenizers in pre-trained large language models (LLMs), which allocate vocabulary based on the corpus distribution at initial training time. As deployment priorities shift and more languages are added, underrepresented languages are fragmented into many tokens per word, increasing decode latency, compute, and energy usage, especially on on-device models where embedding and LM-head matrices form a material cost during decoding. The authors propose an "in-place tokenizer expansion" method for upgrading a pre-trained model's tokenizer under the model producer's control by continuing the original tokenizer's byte pair encoding (BPE) merges on a multilingual corpus. This preserves most original tokens and provides an exact decomposition for new tokens into source tokens. Embeddings for new tokens are initialized as the mean of their source sub-token embeddings. Then a two-stage adaptation—embedding-only training followed by full-model continued pre-training—recovers model quality.

They apply this recipe to an 8B-parameter Mixture-of-Experts model checkpoint (LFM2-8B-A1B) to produce an expanded tokenizer with 128K vocabulary (LFM2.5-8B-A1B). This tokenizer reduces tokens per word drastically for under-tokenized languages like Hindi (2.4× fewer tokens), Vietnamese (2.6× fewer), and Thai (up to 4× fewer), which translates to an estimated 2.2–3.7× decode speedup per character on reference edge devices despite the larger vocabulary size. The expanded model restores or improves benchmark quality on English, code, and multilingual tasks. The paper describes negative experimental findings guiding the design, and releases the model weights and tokenizer. Their approach offers a practical production path for vocabulary upgrades at scale without retraining from scratch or zero-shot transfer.

Key findings

• Continued-BPE expansion yields 63,151/64,400 source tokens carrying over 1-to-1 into the expanded vocabulary of 125,017 active tokens out of 128K nominal size.
• New token embeddings initialized as mean of sub-token embeddings enable stable embedding-only training to recover most of quality lost in zero-shot vocabulary swap.
• Embedding-only training over 600B tokens restores 4.8 aggregate points out of 5.8 points lost initially on an 8B MoE model; full-model continued pre-training over 400B tokens surpasses source quality by +3.6 aggregate points (Figure 3, Table 3).
• Expanded tokenizer encodes Hindi and Vietnamese with 2.4× and 2.6× fewer tokens respectively and Thai with up to 4× fewer tokens than the 65K baseline (Figure 2).
• Estimated 2.2–3.7× per-character decode speed-up on-device for under-tokenized languages when combining token count reduction with measured per-token cost of larger embedding/LM-head layers (Section 7).
• Multilingual Global-MMLU benchmark shows stage 2 performance gains of +7.6 (Hindi), +11.6 (Vietnamese), +9.1 (Indonesian), +6.8 (Malay) points accuracy without regressing previously supported languages (Table 4).
• Training only embedding rows copied from source tokens prevented harmful drift seen when updating frozen parts (Section 8 negative findings).
• Full-model continued pre-training requires a balanced multilingual data mixture to prevent regression on English and code, which dominate source model capabilities (Section 4.2, 6.1).

Threat model

The adversary is an internal system context where the model and tokenizer are controlled by the producer, with no adversarial attack considered. The assumption is that the tokenizer can be upgraded by continuing BPE merges on multilingual corpora under controlled conditions, with access to the original model checkpoint and full training infrastructure for adaptation. There is no threat of adversarial token sequences or unknown model tampering.

Methodology — deep read

The authors start with an 8B-parameter Mixture-of-Experts (MoE) model checkpoint, LFM2-8B-A1B, pretrained with a 65K-token byte-level BPE tokenizer optimized for English, code, and some other languages. Their threat model is a production setting where the model producer fully controls both the original tokenizer and model weights, aiming to upgrade the tokenizer to improve tokenization of previously underrepresented languages without sacrificing existing quality.

They construct a 128K-token expanded tokenizer by initializing the BPE merge table from the original 65K tokenizer and continuing BPE training on a balanced multilingual corpus weighted toward under-tokenized languages (Hindi, Vietnamese, Thai, etc.). Existing merges were fixed, meaning ~63K original tokens carry over one-to-one. New tokens have a canonical decomposition into source tokens (length up to 64).

Embedding initialization creates the new embedding matrix matching the expanded vocabulary size (125,017 active entries). For tokens mapping 1:1 to source tokens, embedding rows are copied unchanged. For newly introduced tokens decomposed into multiple source tokens, embeddings are initialized as the mean of their sub-token embeddings. This also initializes tied output LM-head rows.

Training proceeds in two key stages. Stage 1 only trains the new embedding rows for 600 billion tokens from a multilingual dataset weighted toward the under-tokenized languages to tune the new embeddings with the rest of the model frozen. Stage 2 continues pre-training all parameters for 400 billion tokens on a balanced multilingual mixture covering all languages, enabling the model body to adapt fully to the new vocabulary. They reinitialize optimizer states and warm up learning rates at each stage, using a sequence length of 4,096 tokens. After this two-stage adaptation, the model undergoes the standard mid-training and post-training phases involving long context extensions and instruction tuning.

Evaluation uses a suite of 8 public benchmarks covering knowledge (MMLU-Pro), math (GSM8K, MATH500), code (HumanEval+, LiveCodeBench v5), and multilingual tasks (MGSM, MMMLU). They aggregate these benchmarks to track quality across the four key checkpoints: source with 65K tokens, zero-shot swap to 128K without training, after Stage 1, and after Stage 2. They also analyze per-language accuracy on Global-MMLU (42 languages, 39 with meaningful scores) to track gains for under-tokenized languages without loss for supported languages. They measure tokenizer compression as tokens-per-word ratio on held-out multilingual texts.

For latency, direct side-by-side decode timing is confounded by model differences in releases. Instead, they decouple decode speed per character as the product of tokenization compression (tokens per character) and the per-token decoding cost of the embedding/LM-head matrices, measured by benchmarking the same model body with varying embedding sizes at batch size 1 on representative edge devices. This factorization isolates vocabulary size cost from token count effects.

They release the expanded tokenizer, model weights, and report negative experimental results (e.g. harmful drift when unfreezing copied embeddings prematurely, sensitivity to data mixture in Stage 2). Importantly, continued BPE merges guarantee exact source token decompositions for embedding initialization, which is not available in independent tokenizer unions or zero-shot approaches.

As a concrete example, Hindi’s token count reduces by 2.4×, meaning fewer decoding steps. Initialized new embedding rows per Equation 1 from sub-token centroids are trained with other parameters frozen in Stage 1 to close much of the quality gap, then the entire model is retrained in Stage 2 on balanced data, restoring or exceeding English and code baseline quality. This preserves existing capabilities while significantly improving tokenization and decode efficiency for underrepresented languages.

Technical innovations

• Continued-BPE tokenizer expansion maintaining a single fixed merge table extended on new multilingual data, enabling canonical decomposition of new tokens into source tokens.
• Embedding initialization for new tokens by averaging sub-token embeddings from the original vocabulary, enabling stable training with frozen copied embeddings.
• Two-stage adaptation: embedding-only training with frozen model body followed by full-model continued pre-training with balanced multilingual data to recover and surpass original model quality.
• Latency analysis decoupling token count compression effects from the vocabulary size-induced per-token decode cost, enabling accurate runtime speedup estimation on-device.

Datasets

• Multilingual corpus for tokenizer expansion — size not explicitly stated, but large enough to train continued BPE merges and adaptation — internal Liquid AI corpus weighted toward under-tokenized languages
• Held-out multilingual data (FineWeb 2) plus held-out English, code, JSON samples — used to evaluate tokenizer compression
• Global-MMLU benchmark — ~14,000 multiple-choice questions per language covering 42 languages — public multilingual benchmark
• Eight public benchmarks for aggregate quality: MMLU-Pro, GSM8K, MATH500, GSM-Plus, HumanEval+, LiveCodeBench v5, MGSM, MMMLU

Baselines vs proposed

• Source tokenizer (65K vocab) baseline aggregate quality score: 44.7 / Stage 2 expanded tokenizer (128K vocab) aggregate quality score: 48.3 (+3.6 points improvement) (Table 3, Fig 3)
• Zero-shot vocabulary swap aggregate quality: 38.9 (−5.8 points loss vs source)
• Stage 1 embedding-only training aggregate quality: 43.7 (recovers 4.8 points of initial loss)
• Global-MMLU accuracy on Hindi: Source 32.0%, Stage 2 39.6% (+7.6 points), Vietnamese: Source 35.8%, Stage 2 47.4% (+11.6)
• Decode speedup estimate for under-tokenized languages: 2.2× to 3.7× per-character speedup on edge devices versus source tokenizer encoding
• Token count per word for Thai reduced 4×, for Vietnamese 2.6×, and for Hindi 2.4× compared to source tokenizer (Figure 2)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.15232.

Fig 1: Tokenizer expansion pipeline. Top: the expanded tokenizer is constructed by continuing the source

Fig 2 (page 1).

Limitations

• The training corpus for adapter stages and tokenizer expansion is internal and proprietary, limiting reproducibility.
• Evaluation covers mainly high-resource and some under-tokenized languages; underrepresented languages like Thai lack direct multilingual benchmark evaluation.
• Latency estimates are synthesized from separate token-count compression and per-token decode costs rather than direct end-to-end timing on identical checkpoints.
• Continued pre-training requires very large compute (hundreds of billions of tokens) not feasible for all projects.
• The approach assumes control over the tokenizer design and pre-trained checkpoint, limiting applicability when swapping third-party tokenizers.
• Negative findings and ablations on alternative embedding initializations and data mixtures suggest sensitivity but these are not exhaustively explored.

Open questions / follow-ons

• How well does tokenizer expansion perform on truly low-resource or rare languages outside the multilingual corpus used for continued training?
• Can more sophisticated embedding initialization beyond simple averaging, tailored to semantic or syntactic embeddings, accelerate or improve quality recovery?
• What are the minimal continued pre-training token budgets required for stable adaptation across different model scales and architectures?
• How does tokenizer expansion interact with downstream instruction tuning, prompting, or fine-tuning for task-specific uses beyond pre-training benchmarks?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this paper presents a practical method to efficiently extend multilingual vocabulary coverage in on-device LLMs without retraining from scratch or sacrificing performance on existing languages. The continued-BPE tokenizer expansion approach reduces token fragmentation on underrepresented languages, thereby significantly improving decode latency and compute efficiency—critical for real-time CAPTCHA generation or verification with multilingual user bases on resource-constrained devices.

Additionally, the detailed analysis of embedding initialization and staged adaptation can inform similar vocabulary upgrade challenges in CAPTCHA-related ML models, where balancing tokenization granularity against latency and accuracy is crucial. The latency decomposition separating tokenizer compression from vocabulary size cost is valuable for predicting decode time trade-offs under deployment constraints. However, the method’s reliance on controlled tokenizer and checkpoint access limits direct application if using third-party models or tokenizers out-of-the-box. Nonetheless, the insights and recipe provide important foundations for maintaining and evolving multilingual on-device language models deployed in bot-defense contexts.

Cite

@article{arxiv2607_15232,
  title={ In-Place Tokenizer Expansion for Pre-trained LLMs },
  author={ Jimmy T. H. Smith and Tarek Dakhran and Alberto Cabrera and Simon S. Lee and Paul Pak and Aditya Tadimeti and Tim Seyde and Maxime Labonne and Alexander Amini and Mathias Lechner },
  journal={arXiv preprint arXiv:2607.15232},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.15232}
}

Read the full paper

• arXiv:2607.15232 (abstract)
• arXiv:2607.15232 (PDF)