ReToken: One Token to Improve Vision-Language Models for Visual Retrieval

Source: arXiv:2607.28627 · Published 2026-07-30 · By Yao Xiao, Reuben Tan, Zhen Zhu, Yuqun Wu, Jianfeng Gao, Derek Hoiem

TL;DR

This paper addresses the challenge of long visual context in vision-language models (VLMs), where performance degrades as distractors increase and full-context processing is costly under GPU memory constraints. The authors identify that conventional attention-based retrieval is inefficient due to weak correlation between query-key attention scores and relevance. Instead, they propose ReToken, a single learnable embedding token trained to retrieve query-relevant visual tokens by scoring cosine similarity in the value-projection space rather than the traditional query-key space. This lightweight addition to frozen VLMs enables more precise sparse selection of relevant visual frames or images, improving visual retrieval accuracy without costly full context processing.

ReToken achieves substantial gains across diverse benchmarks, including a 13.4-point (20% relative) accuracy improvement on Visual Haystacks with Qwen3VL-8B and 12.4 points on InternVL3.5. Remarkably, although trained only on multi-image QA datasets, it transfers zero-shot to long-video benchmarks (LVBench) yielding an 8.0-point gain, demonstrating strong generalization for long-range multimodal reasoning. The method trains efficiently on a single H100 GPU and allows long-video inference to fit within memory constraints by caching visual key-value embeddings and applying a two-pass retrieval-then-answer pipeline. Extensive ablations confirm that retrieval in the value space, a single token design, and a retrieval-focused loss are key factors in ReToken’s success.

Key findings

• Attention-based retrieval achieves only 5.1% average recall@1 on long-video QAEgo4DTest-MC, while ReToken boosts recall substantially (Fig. 1b).
• Value-based retrieval scoring improves Recall@1 from 65.7% to 78.0% on Qwen3VL and from 78.8% to 83.8% on InternVL3.5 in 2-image retrieval (Tab. 1).
• ReToken improves Qwen3VL-8B accuracy on Visual Haystacks by +13.4 points at context size C=50, corresponding to >20% relative gain (Tab. 4).
• On LVBench long-video benchmark, ReToken yields +8.0-point accuracy gain zero-shot with Qwen3VL-8B, although trained only on multi-image QA data (Tab. 6b).
• ReToken retrieval pass attends to up to 256 frames per early transformer layer to scale long video retrieval within GPU memory, enabling efficient inference (Sec. 3.2).
• Partial fine-tuning of first 3 VLM layers in addition to ReToken yields up to +10.8 points gain on Visual Haystacks (Tab. 4), but degrades video QA performance (Tab. 8).
• Retrieval based on average image values consistently outperforms retrieval based on keys, with recall 64.7% vs 59.4%, accuracy 72.0% vs 70.6% (Tab. 7).
• ReToken outperforms all baselines including ReKV, SigLIP2, CoT and external MIRAGE retrieval across varying context sizes and retrieval budgets (Tab. 2 and Fig. 5).

Threat model

N/A — the paper centers on improving retrieval and inference efficiency in vision-language models rather than defending against malicious adversaries. The retrieval problem is framed as selecting relevant visual tokens from large, noisy input contexts under resource constraints.

Methodology — deep read

Threat Model & Assumptions: The adversary is not explicitly modeled; the focus is on improving retrieval precision for relevant visual content in VLMs under long-context conditions where visual distractors overwhelm retrieval. The VLM is pretrained and frozen by default; only a learnable retrieval token and projection matrix are trained. The model assumes access to labeled relevance for multi-image QA retraining.

Data: Training uses the MIRAGE multi-image question-answering dataset (95% train, 5% validation splits), combining RetVQA, SlideVQA, WebQA, and synthetic LLaVA Visual Instruct data with distractors. Labels provide binary relevance per image to the query. Evaluation uses Visual Haystacks (1,000 QA pairs with distractors), QAEgo4DTest-MC (long egocentric video QA with annotated relevant segments), LVBench (103 YouTube videos averaging 68 minutes), and Video-MME (900 videos with short, medium, long splits).

Architecture / Algorithm: ReToken appends a single learnable embedding token Xr to question tokens. At the final transformer layer (LN) of the frozen VLM, each frame's mean value vector is computed by averaging value projections (v_i^N) from the visual tokens belonging to that frame. A learnable projection Wr maps the output retrieval token embedding Xr^(N) to Zr in the same space. Retrieval scores for each frame are the cosine similarity between Zr and frame mean values. The top-K frames with highest scores are selected.

Training paradigm is frozen VLM weights, updating only Xr and Wr using a class-balanced binary cross-entropy retrieval loss Lret with learnable temperature τ. Optionally, partial fine-tuning of early VLM layers is done with a combined retrieval plus generation loss to preserve QA performance.

Training Regime: Training on a single H100 GPU for 3 epochs (Qwen3VL-8B) or 1 epoch (InternVL3.5-8B) with batch size 64. Learning rate 3e-4 for frozen VLM setting, 2e-5 with partial fine-tuning. Linear warmup and cosine decay.

Evaluation Protocol: Retrieval performance measured by recall@1, accuracy of answer generation post retrieval, and comparison to baselines including attention-based ReKV, SigLIP2, and CoT prompting. Ablations across retrieval budgets (K), context sizes (C), and transformer layers. Video inference uses a two-pass approach: first pass computes retrieval scores with limited visual token attention budget (K1=256 frames per early layer) using cached KV states, second pass generates answers attending only to selected frames.

Reproducibility: Code is publicly released at the authors’ GitHub. The MIRAGE training dataset is partially public; some multi-image QA components are synthetic or aggregated from prior datasets. Pretrained VLMs Qwen3VL-8B and InternVL3.5-8B are referenced from prior works. Detailed training and inference hyperparameters are shared.

Concrete Example End-to-End: Given a long video broken into F frames, each frame represented by M/F visual tokens, these tokens are embedded and cached once in the VLM’s KV cache. For a query, the retrieval token Xr is appended and passed through the frozen model to produce a projected embedding Zr. Cosine similarity scores with each frame’s mean value vector are computed; the top-K frames are selected. During answer generation, only the KV cache entries for these frames are loaded and attended to, yielding more efficient inference and improved accuracy due to focused retrieval on relevant visual context rather than noisy global attention.

Technical innovations

• Demonstration that retrieval in the value projection space, rather than query-key attention space, provides much stronger relevance signals for visual token retrieval.
• Introduction of a single learnable retrieval token embedding trained with explicit retrieval supervision to replace noisy question-averaged queries.
• A two-pass retrieval-then-generation pipeline using cached key-value pairs to enable scalable long-video inference within GPU memory limits.
• Partial fine-tuning of early transformer layers in combination with the retrieval token to achieve cleaner cached visual representations and improved retrieval precision.

Datasets

• MIRAGE multi-image QA dataset (aggregated from RetVQA, SlideVQA, WebQA plus synthetic LLaVA Visual Instruct 150K) — training set for retrieval token
• Visual Haystacks — 1,000 question-answer pairs with distractor images derived from COCO — evaluation
• QAEgo4DTest-MC — egocentric video QA with temporal relevance annotations, videos 4-20 mins — evaluation
• LVBench — 103 public YouTube videos totaling ~117 hours average 68 min length — evaluation
• Video-MME — 900 videos with 2,700 QA pairs partitioned into short, medium, long splits — evaluation

Baselines vs proposed

• GT Cache (oracle): Recall@1 = 100%, Accuracy C=50 = 80.7%; ReToken: Recall@1 = 64.7%, Accuracy = 72.0%
• SigLIP2: Accuracy = 60.3% at C=50; ReToken: Accuracy = 72.0% at C=50 on Visual Haystacks (Tab. 2,4)
• ReKV attention-based retrieval: Recall@1 = 20.8%, Accuracy = 60.7% at C=50; ReToken: Recall@1 = 64.7%, Accuracy = 72.0%
• CoT retrieval: Accuracy = 51.5% at C=50; ReToken: Accuracy = 72.0%
• Zero-shot transfer to LVBench long videos: Qwen3VL-8B baseline accuracy = 40.6%, + ReToken = 48.6 (+8.0 points)
• Partial fine-tuning + ReToken increases Recall@1 to 68.3% and accuracy to 75.0% at C=50, surpassing frozen setting (Tab. 8a).

Limitations

• ReToken is trained only on multi-image QA datasets, not on diverse video QA datasets, so video domain long-term temporal retrieval may be suboptimal.
• Partial fine-tuning benefits image retrieval but degrades video QA accuracy, indicating a possible trade-off between clean cached representations and temporal modeling.
• Evaluation focuses on multiple-choice and yes/no QA metrics; open-ended generation or other multimodal tasks are not explored.
• Memory savings and inference speedups are discussed but detailed latency and scalability benchmarks with extremely large contexts (>100K frames) are not reported.
• The retrieval relies on cosine similarity and mean pooling over frame values, which may be less effective if frame-level granularity or object-level localization is required.
• No explicit adversarial analysis of retrieval robustness against distractor manipulation or query paraphrasing was presented.

Open questions / follow-ons

• Can ReToken be extended to hierarchical or multiple retrieval tokens to capture more complex visual-language interactions or multi-entity queries?
• How does ReToken perform under adversarial distractors or queries deliberately designed to confuse retrieval?
• Would joint training of ReToken on combined image and long-video QA datasets improve retrieval and transfer robustness further?
• Can the approach be adapted for open-ended generative tasks beyond multiple-choice QA, such as captioning or dialog over long videos?

Why it matters for bot defense

Bot defenses and CAPTCHAs sometimes leverage vision-language models to interpret complex visual contexts or video streams when challenged. ReToken’s approach to efficiently retrieving relevant visual tokens from long, distractor-heavy inputs without exhaustive processing addresses a key scalability bottleneck. Its design—using a lightweight learnable token trained explicitly for retrieval in the value space—offers a practical method to focus model attention on critical visual elements, improving accuracy and reducing computational load. Practitioners designing multi-modal CAPTCHA challenges or bot detection systems could consider similar retriever-token mechanisms to handle long video or image sequences under resource constraints. Moreover, the two-pass retrieval generation setup with cached visual KV pairs suggests a way to handle repeated queries efficiently, relevant to interactive or adaptive CAPTCHA scenarios where latency and GPU memory are constrained.

Cite

@article{arxiv2607_28627,
  title={ ReToken: One Token to Improve Vision-Language Models for Visual Retrieval },
  author={ Yao Xiao and Reuben Tan and Zhen Zhu and Yuqun Wu and Jianfeng Gao and Derek Hoiem },
  journal={arXiv preprint arXiv:2607.28627},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.28627}
}

Read the full paper

• arXiv:2607.28627 (abstract)
• arXiv:2607.28627 (PDF)