Bekko Embedding: Parameter-Efficient Multilingual Retrieval with Ultra-Compact Encoders

Source: arXiv:2607.25180 · Published 2026-07-28 · By Yuichi Tateno

TL;DR

This paper addresses the problem of building highly parameter-efficient multilingual dense retrieval models that maintain competitive retrieval quality at dramatically reduced inference cost. The authors introduce Bekko Embedding, a family of ultra-compact encoders obtained by aggressive structural layer pruning of a 22-layer mmBERT-small multilingual encoder down to as few as 4 or 13 layers, yielding models with only 7.7M and 25M active (non-embedding) parameters respectively. These models are trained via a two-stage contrastive learning pipeline using an unprecedentedly large public multilingual corpus with 1.1 billion pairs, including synthetic queries from two complementary large language model synthesizers, followed by hard-negative mining and fine-tuning. Despite the aggressively reduced model size, Bekko matches or outperforms much larger models (with 20–40x more active parameters) on official multilingual retrieval benchmarks like MMTEB Multilingual v2. On retrieval quality (nDCG@10), the smallest Bekko-a8m model scores 56.2, surpassing multilingual-e5 and BGE-M3, while the larger a25m model reaches 57.5, comparable to gte-multilingual-base. The 384-dimensional embeddings enable efficient downstream similarity search. Importantly, the small model size translates into significantly faster CPU and GPU inference — a8m runs 1.6x faster than multilingual-e5-small on x86 CPUs and is even deployable on resource-constrained devices like Raspberry Pi 5 and in-browser. The authors release all model weights, the entire stage-1 training corpus, and independently mined stage-2 hard negatives to encourage reproducible research and further progress in parameter-efficient multilingual retrieval.

Key findings

• bekko-embedding-v1-a8m (7.67M Active Parameters) achieves 56.2 nDCG@10 on MMTEB Multilingual v2 Retrieval, outperforming multilingual-e5 family and BGE-M3 models with 20–40x higher AP.
• bekko-embedding-v1-a25m (24.93M AP) reaches 57.5 nDCG@10, comparable to gte-multilingual-base (≈113M AP).
• Both models support up to 8192-token inputs, enabling state-of-the-art long-input retrieval performance; on NanoLongEmbed, a25m is the strongest dense model compared.
• Two-stage contrastive training on 1.1 billion pairs from a publicly released multilingual corpus (100+ languages), combining doc_doc and query_doc pairs along with synthetic queries from two LLM-based synthesizers.
• Structural layer pruning of mmBERT-small from 22 to 4 or 13 layers preserves key multilingual retrieval knowledge without distillation, yielding ultra-compact models.
• The a8m model runs 1.6× faster than multilingual-e5-small on x86 CPUs and 1.5× on GPU under identical conditions, and is the fastest on Raspberry Pi 5.
• Vocabulary embedding matrix quantized with row-wise int8 compression reduces a8m model size to 124 MiB (about 1/3 of fp32 size) enabling browser inference via Transformers.js.
• Output embedding dimension is 384 (truncatable to 256/128/64) reducing downstream indexing/search costs compared to 1024+ dimensions in larger models.

Threat model

Adversary is a general-purpose user or automated system issuing queries to a dense retrieval system potentially on-device or on CPU-only servers. The attacker cannot tamper with model parameters or training, nor launch adversarial input attacks explicitly studied. The focus is on efficient, low-latency retrieval against realistic query distributions rather than adversarial scenarios.

Methodology — deep read

The authors start with a threat model focused on practical inference environments: resource-limited CPUs, edge devices, and browsers, where latency and memory constrain applicability. The adversary is standard IR noise; no adversarial attack is modeled. The core methodological contribution is minimizing Active Parameters (AP) — the non-embedding parameters that govern inference FLOPs — by pruning a pretrained 22-layer mmBERT-small encoder down to 4 or 13 layers. This structural pruning removes entire layers, preserving pretrained weights without teacher-student distillation or continued pretraining. The layer selection scheme keeps early contiguous layers plus at least one deep Global attention layer (layer 18) to maintain long-range context modeling. This yields pruned base models (7.7M and 24.9M AP) which then serve as initialization for contrastive fine-tuning.

For data, the authors compile a massive multilingual corpus of 1.1B+ pairs covering 100+ languages, publicly released as bekko-embedding-v1-unsupervised. It integrates various pair types: doc_doc (symmetric pairs such as translations or related paragraphs) and query_doc (asymmetric pairs resembling search queries or titles related to documents). The corpus combines many sources: NLLB and CCMatrix parallel bitext, Wikipedia subsets, multilingual news, and existing IR benchmarks. Additionally, two complementary LLM-based query synthesizers generate 160M+ synthetic pairs: a high-quality setup using Qwen3.5-35B-A3B for in-depth English domains, and a faster multilingual query generator (Qwen3-1.7B based) producing multiple query forms over 21 languages.

Training follows a two-stage contrastive learning regime. Stage 1 is large-scale contrastive training on the 1.1B public pairs, using a masked contrastive loss whose direction depends on pair type (query_doc vs doc_doc), combined with a Matryoshka objective for multi-scale representation learning. Stage 2 is hard-negative fine-tuning where the model is trained with mined hard negatives (released publicly). Batches are drawn from single data subsets with no mixing to maintain homogeneous in-batch negatives. No knowledge distillation or external teacher model is used.

The model architecture remains ModernBERT style: encoders with global-local attention patterns repeated every 3 layers. Pruned models preserve this rhythm, e.g., layer 18 is a deep global layer kept instead of the final layer 21. The output dimensionality is 384, truncated down to 64 in some experiments. Vocabulary embedding matrices are row-wise quantized to int8 to reduce file size. All training and experiments run on a single GPU; training speed is approx. 3 days for a8m, 8 days for a25m.

Evaluation uses comprehensive multilingual retrieval benchmarks including MMTEB Multilingual v2 Retrieval (official metric: nDCG@10), Multilingual NanoBEIR, and NanoLongEmbed for long-document retrieval, plus author-developed HAKARI-Bench for development. Baselines include strong larger-scale models such as multilingual-e5, BGE-M3, Granite R2, gte-multilingual-base, and EmbeddingGemma, using an active parameter (AP) axis for fair comparison of inference compute. Efficiency and inference latency/speed experiments include x86 CPUs, GPUs, Raspberry Pi 5, and in-browser evaluation via Transformers.js.

Reproducibility is high: all model weights, the entire stage-1 training corpus, and the stage-2 hard negatives are released publicly. Source code for training procedure is implied but not explicitly mentioned. The paper’s ablation studies thoroughly analyze pruning patterns, data design, and training loss components with quantitative results.

Technical innovations

• Systematic use of Active Parameters (non-embedding parameters) as the primary metric for model efficiency and scaling in multilingual dense retrieval.
• Distillation-free structural layer pruning of mmBERT-small from 22 layers to ultra-compact 4 or 13-layer encoders retaining key global attention layers for long-context modeling.
• Two-stage contrastive training on a massive publicly released corpus of over one billion multilingual text pairs combining doc-doc and query-doc types, leveraging complementary LLM-based synthetic query generation.
• Masked contrastive loss with directionality depending on pair type combined with Matryoshka objectives enabling effective training without teacher distillation.
• Row-wise int8 quantization of large vocabulary embedding matrices reducing model file size substantially without affecting Transformer computation.

Datasets

• bekko-embedding-v1-unsupervised — ~1.1 billion pairs — public corpus compiled from multilingual bitext, Wikipedia, news, synthetic queries
• bekko-embedding-v1-hard-negatives — 759,587 rows — independently mined hard negatives for fine-tuning
• Multilingual NanoBEIR — 14 languages — public benchmark datasets (condensed from BEIR)
• MMTEB Multilingual v2 Retrieval — official public multilingual retrieval benchmark

Baselines vs proposed

• multilingual-e5-small: nDCG@10 = ~53.5 vs bekko-a8m 56.2 on MMTEB Multilingual v2
• BGE-M3: nDCG@10 < bekko-a8m 56.2 despite 40× AP
• gte-multilingual-base: nDCG@10 ~57.5 vs bekko-a25m 57.5 (parity)
• Granite Embedding R2 small (28M AP): retrieval quality lower than bekko-a25m (25M AP)
• bekko-a8m CPU inference speed: 1.6× faster than multilingual-e5-small on x86 CPU

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.25180.

Fig 1: Parameter efficiency of multilingual embedding models: retrieval quality (HAKARI-Bench

Fig 2: Bekko’s construction pipeline. Without distillation, the pruned model serves as the base for

Fig 3: Structural layer pruning. We adopt designs that add the deep Global layer (18), not the final

Fig 4: Retrieval quality versus AP (log scale; the 12 dense models of the unified set; 2×2 panels)—the

Fig 5: Long-input retrieval (NanoLongEmbed) and long-document retrieval (NanoMLDR). BM25

Fig 6: Distribution-size breakdown and the effect of embedding int8. The compression target is the

Fig 7: Retrieval quality versus CPU inference speed (GPU-less setting; the models of the unified set

Fig 8: Output-vector compression (dimensionality reduction × quantization; HAKARI-Bench Overall).

Limitations

• Low-resource languages performance remains limited due to skew in training data distribution toward English and major languages.
• No adversarial robustness or attack evaluation reported; robustness to malicious queries or attacks unknown.
• Pruning choice based on heuristics; no learned pruning or formal layer importance analysis presented.
• Evaluation focused on retrieval accuracy and latency; no end-to-end search quality or user-centered metrics.
• All training done on a single GPU; replication might be constrained by access to large-scale 1.1B pair data corpus.
• While vocabulary embedding compression reduces size, actual model storage and memory footprint on extremely constrained devices remains to be fully characterized.

Open questions / follow-ons

• How far can performance be pushed with further pruning or compression combined with knowledge distillation or adapter modules?
• What are the trade-offs between pruning patterns and long-context retrieval quality beyond empirically selected layers?
• Can the approach generalize to larger hidden sizes or other pretrained multilingual encoders beyond mmBERT-small?
• How robust are the models to noisy, adversarial, or out-of-distribution queries encountered in real-world retrieval?

Why it matters for bot defense

This work presents a clear path to highly efficient multilingual dense retrieval models that operate with very small active parameter counts, enabling fast on-device or low-resource CPU inference. For bot-defense and CAPTCHA systems that require quick similarity search over multilingual text (for example, query verification, user-generated content indexing, or multilingual intent detection), Bekko’s ultra-compact encoders could enable low-latency semantic retrieval at the edge or even in-browser without cloud roundtrips. The pruning and training recipes demonstrate that a relatively modest GPU budget suffices to produce competitive models without large-scale distillation or ensemble dependencies. Practitioners can leverage Bekko’s open datasets and weights to build or fine-tune compact models for real-time multilingual retrieval tasks embedded within security or bot-detection pipelines, where inference speed and footprint are critical constraints. However, care should be taken since low-resource language robustness and adversarial behavior remain untested.

Cite

@article{arxiv2607_25180,
  title={ Bekko Embedding: Parameter-Efficient Multilingual Retrieval with Ultra-Compact Encoders },
  author={ Yuichi Tateno },
  journal={arXiv preprint arXiv:2607.25180},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.25180}
}

Read the full paper

• arXiv:2607.25180 (abstract)
• arXiv:2607.25180 (PDF)