ViQ: Text-Aligned Visual Quantized Representations at Any Resolution

Source: arXiv:2606.27313 · Published 2026-06-25 · By Xumin Yu, Zuyan Liu, Zhenyu Yang, Yuhao Dong, Shengsheng Qian, Jiwen Lu et al.

TL;DR

This paper addresses the challenge of creating a unified discrete representation for both text and vision that balances semantic richness and low-level visual detail. Existing discrete visual tokenization methods tend to either lose semantic information (focusing only on reconstruction) or lose fine-grained details (focusing only on semantics). ViQ (Visual Quantized Representations) proposes a two-stage training framework to tackle this: first, a text-aligned pre-training stage that jointly trains a visual encoder with supervision from a pretrained large language model to produce semantically rich, native-resolution continuous features; second, a quantization stage that compresses these features into discrete tokens using proximal representation learning and a novel position-aware multi-head finite scalar quantization. This design ensures flexible input resolutions and preserves both semantics and details in discrete codes. The resulting ViQ framework yields visual encodings competitive with state-of-the-art continuous encoders on multiple multimodal benchmarks while offering significant training efficiency gains and strong image reconstruction quality.

Experiments demonstrate ViQ achieves comparable or superior performance to major continuous visual encoders such as InternViT-2.5 and SigLIP2 across nine benchmarks including VQA, OCR, and document understanding tasks. It also substantially outperforms previous quantized models like QLIP and UniTok by large margins. Furthermore, ViQ reduces training iteration time by 20%-70% using pre-extracted discrete vision codes. Reconstructions from ViQ codes yield a PSNR of 22.73 and an rFID of 0.62 on ImageNet, surpassing prior discrete tokenizers. Overall, ViQ offers a practical discrete visual encoding compatible with large language models that does not sacrifice detail or semantic alignment, enabling efficient large-scale multimodal training and storage.

Key findings

• ViQ achieves an average multimodal understanding score of 57.2 (Qwen2.5-1.5B backbone) and 63.9 (Qwen2.5-7B) across 9 benchmarks, surpassing prior SOTA continuous encoders (InternViT-2.5 at 57.0 and 63.8 respectively) under comparable visual encoder parameter counts (~1.3B).
• ViQ compresses raw images into discrete codes storing only 1/96th of the original space, enabling highly efficient image storage compared to JPEG at similar or better reconstruction quality.
• Image reconstruction metrics on ImageNet-1K show ViQ achieves PSNR=22.73, SSIM=0.66, and rFID=0.62, outperforming prior discrete tokenizers like UniTok (PSNR=25.32, rFID=0.37) in balancing semantics and reconstruction quality.
• In training speed evaluations with Qwen2.5 models (0.5B to 7B), ViQ obtains forward pass speedups of 46%-78% and full training iteration speedups of 20%-70% compared to SigLIP2-g continuous visual encoder.
• ViQ’s proximal representation learning and multi-head finite scalar quantization significantly reduce information loss during feature discretization compared to direct quantization approaches.
• ViQ supports native input image resolutions dynamically via resized positional embeddings and 2D rotary position encoding, unlike many fixed-resolution baselines.
• Text-aligned pre-training with language supervision and self-distillation preserves semantic fidelity in the visual encoder embeddings across resolutions.
• ViQ outperforms other quantized visual encoders (QLIP, UniTok) by large margins on multimodal tasks, particularly on text- and document-heavy benchmarks requiring fine-grained visual understanding.

Methodology — deep read

1. Threat Model & Assumptions: The work does not explicitly define a threat model but assumes a need for multimodal vision-language models to represent images discretely with minimal information loss for downstream tasks. The adversary, if any, is not addressed; the paper focuses on balancing semantic and low-level detail retention rather than adversarial robustness.

2. Data: Training uses large-scale multimodal datasets consisting of image-text-answer triplets. The main fine-tuning and evaluation datasets include 2 million samples from LLaVA-OneVision, covering diverse multimodal benchmarks such as MMStar, MMMU, SimpleVQA, InfoVQA, TextVQA, DocVQA, OCRBench, AI2D, and ChartQA. ImageNet-1K validation set (256×256 crop) is used for reconstruction evaluation. Training input images are progressively scaled up from low resolution to native resolution maintaining aspect ratios.

3. Architecture / Algorithm: ViQ consists of a visual encoder (based on pretrained SigLIP2-g, output dim C=1536), followed by bottleneck compression layers reducing dimensionality to D=128 and then d=6. A proximal representation learning stage applies L-infinity norm constraints to compress latent features onto a hypercube, reducing feature space complexity before quantization. Quantization uses multi-head Finite Scalar Quantization (FSQ) that discretizes the latent features into integer codes. A position-aware mechanism using 2D rotary positional embeddings encodes spatial resolution information explicitly. The quantizer expands each patch into 4 smaller patches (2×2) processed with multi-head self-attention before quantization. Text-aligned pretraining uses supervision from a frozen large language model computing cross-entropy loss on predicted answers. A self-distillation loss encourages semantic token alignment between fixed-resolution teacher and any-resolution student encoders. Low-level reconstruction is supervised using negative log-likelihood on VAE latent space.

4. Training Regime: Stage 1 pre-training (text-aligned) uses 128 NVIDIA A100 GPUs, progressively increasing image resolution and sequence length, optimizing a combined loss of text cross-entropy and distillation. Stage 2 quantized training uses 256 A100 GPUs, jointly optimizing text loss, distillation loss, and reconstruction loss from a pretrained VAE latent space. Hyperparameters like quantization levels (8,8,8,5,5,5) yield a total codebook size of 64,000. Qwen2.5-VL-0.5B serves as language model supervision. Details on optimizer, epochs and batch size are referenced in appendix but not fully detailed in the main text.

5. Evaluation Protocol: The authors evaluate multimodal understanding accuracy across 9 diverse benchmarks, comparing to strong continuous visual encoders (InternViT-2.5, AIMv2, SigLIP2-g, OryxViT) and quantized encoders (QLIP, UniTok). Reconstruction quality is evaluated via PSNR, SSIM, and rFID on ImageNet-1K. Training efficiency is measured as average forward pass and full iteration step times on single-node setups for different Qwen2.5 model sizes and training data sizes (4k and 16k samples). Ablation studies test training losses and quantization mechanisms. Some baseline results are reproduced using official checkpoints with consistent evaluation tools (LMMs-Eval).

6. Reproducibility: Code and pretrained weights are publicly released on GitHub and HuggingFace (links provided). The datasets used for fine-tuning and evaluation are standard or publicly available benchmarks. Details about training recipes and full hyperparameters are in the appendix.

Concrete Example: For a single input image at native resolution, ViQ’s encoder first extracts continuous visual features at dimension C=1536. These are compressed via bottleneck layers to D=128 then d=6. The compressed features are constrained onto a hypercube L-infinity norm space (∥f1∥∞=1) as a proximal representation. Next, a multi-head FSQ quantizes the features into discrete tokens, expanding each patch into 4 via 2×2 self-attention to improve representational capacity. Position embeddings with 2D RoPE encode spatial location. These discrete tokens are used as input to the large language model with textual queries for multimodal understanding, supervised by text-answer cross entropy combined with distillation loss from fixed resolution teacher. The full training optimizes both semantic alignment and low-level feature reconstruction. Once trained, the ViQ quantized codes can be stored efficiently and also decoded to reconstruct the image with high fidelity.

Technical innovations

• Two-stage training framework combining text-aligned pre-training to enhance semantic alignment with proximal representation learning and position-aware multi-head finite scalar quantization for detailed, native-resolution discrete visual tokens.
• Proximal representation constraining continuous visual features onto an L-infinity norm hypercube prior to quantization to minimize precision loss and improve quantization stability.
• Multi-head finite scalar quantization expands and attends over visual patches, enabling flexible tokenization of native-resolution images into discrete codes with richer representational capacity.
• 2D rotary position embeddings adapted for vision tokens to encode spatial location information dynamically at arbitrary input resolutions, overcoming fixed positional embedding limitations.

Datasets

• LLaVA-OneVision — 2,000,000 samples — multimodal image-text-answer triplets used for fine-tuning and evaluation across multiple benchmarks
• ImageNet-1K Validation — standard 256x256 images — used for image reconstruction evaluation
• MMStar, MMMU, SimpleVQA, InfoVQA, TextVQA, DocVQA, OCRBench, AI2D, ChartQA — subsets or benchmarks drawn from public sources for multimodal evaluation

Baselines vs proposed

• InternViT-2.5 (0.3B params): average multimodal score = 56.5 vs ViQ (1.3B params): 57.2 (Qwen2.5-1.5B backbone)
• InternViT-2.5-6B: average score = 57.0 vs ViQ: 57.2
• InternViT-2.5-6B: avg score = 63.8 vs ViQ: 63.9 (Qwen2.5-7B backbone)
• SigLIP2-g (1.1B params): avg score = 53.1 vs ViQ: 57.2 (Qwen2.5-1.5B)
• QLIP (0.3B params): avg score = 29.7 vs ViQ: 57.2
• UniTok (0.3B params): avg score = 33.0 vs ViQ: 57.2
• On ImageNet reconstruction: UniTok PSNR=25.32, rFID=0.37 vs ViQ PSNR=22.73, rFID=0.62
• Training speed (Qwen2.5 0.5B) forward pass: SigLIP2-g baseline vs ViQ yields 70-78% speedup
• Training iteration step time (Qwen2.5 7B): SigLIP2-g baseline vs ViQ yields 20-40% speedup

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.27313.

Fig 1: ViQ delivers high-quality multimodal quantized representations with both high-

Fig 2: Approach of ViQ Representation Learning. Stage 1 enables multimodal alignment

Fig 4: Representing images with ViQ. We show the image compression and visual reconstruction

Fig 3: Comparisons on Training Efficiency Across Different Visual Encoders. We conduct the

Fig 5 (page 10).

Fig 6 (page 10).

Fig 7 (page 10).

Fig 8 (page 10).

Limitations

• ViQ, as a quantized discrete representation, still shows residual performance gaps on detail-heavy tasks such as OCRBench compared to some continuous encoders due to compression artifacts.
• The approach focuses on fidelity and semantic alignment rather than adversarial robustness or security evaluation of representations.
• Quantization codebooks and hyperparameters need careful tuning; generalization beyond tested benchmarks or extremely high resolutions is not fully characterized.
• Training requires large-scale compute (128-256 A100 GPUs) which may limit accessibility for smaller groups.
• The complexity of multi-stage training and combination of losses adds engineering overhead.
• Loss of some very high-frequency image details after quantization may limit applications that require ultra-fine visual nuances.

Open questions / follow-ons

• Can multi-scale or residual quantization strategies further reduce the detail loss inherent in discrete tokenization?
• How well does ViQ generalize to other modalities or more diverse vision-language tasks beyond those benchmarked?
• What are the trade-offs in downstream robustness or adversarial susceptibility introduced by quantized versus continuous representations?
• Can lightweight or distilled versions of ViQ maintain performance benefits while reducing computational costs?

Why it matters for bot defense

ViQ demonstrates that discretized visual representations can effectively preserve both semantic understanding and fine-grained details at native image resolutions. For bot-defense and CAPTCHA engineering, this work suggests promising directions for encoding challenging visual puzzles or image-based tokens into discrete tokens aligned with language modalities. By enabling higher efficiency in training and inference through discrete codes, CAPTCHA systems integrating large multimodal models could potentially handle more complex visual tasks with reduced latency and resource use. The position-aware quantization and multi-resolution support also hint at avenues to design CAPTCHAs that are both human-friendly and model-resistant by exploiting subtle low-level visual details. However, the residual trade-off between compression and detail retention means that outright replacement of continuous feature extractors with quantized counterparts should be carefully benchmarked, especially for security-critical or adversarial CAPTCHA contexts. Furthermore, the two-stage training methodology may inspire improved pretraining pipelines for visual tokenizers tailored to CAPTCHA schemes, embracing joint semantic and reconstruction losses to robustly encode visual cues relevant for distinguishing human versus bot inputs.

Cite

@article{arxiv2606_27313,
  title={ ViQ: Text-Aligned Visual Quantized Representations at Any Resolution },
  author={ Xumin Yu and Zuyan Liu and Zhenyu Yang and Yuhao Dong and Shengsheng Qian and Jiwen Lu and Han Hu and Yongming Rao },
  journal={arXiv preprint arXiv:2606.27313},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.27313}
}

Read the full paper

• arXiv:2606.27313 (abstract)
• arXiv:2606.27313 (PDF)