Scalable Visual Pretraining for Language Intelligence

Source: arXiv:2607.09657 · Published 2026-07-10 · By Yiming Zhang, Zhonghan Zhao, Wenwei Zhang, Haiteng Zhao, Tianyang Lin, Yunhua Zhou et al.

TL;DR

This paper addresses the limitations of traditional text-only pretraining for large language models (LLMs) by introducing Visual Pretraining (VP), a novel approach that leverages raw document images directly without extracting or converting text. The key insight is that many types of knowledge, especially in scientific documents, are visually encoded through equations, figures, tables, and layout, which text extraction methods distort or lose. VP trains foundation models to predict the next visual latent tokens in these images, integrating native visual structure into the learning process. Empirically, VP consistently outperforms matched text-only pretraining across multiple modern backbones (Qwen 3.5/3, Llama 3.2 Vision/3.1) on various scientific reasoning benchmarks such as GPQA, AIME-25, MMLU-Pro, and HLE. These gains are especially pronounced on visually dense documents and scale efficiently, achieving stronger results with fewer training tokens and smaller model sizes. Additionally, VP improves cross-modal alignment and multimodal benchmark performance without paired image-text supervision, demonstrating emergent multimodal capabilities. Attention analyses further indicate that VP enhances the model's ability to focus on semantically relevant visual regions correlating with reasoning cues. Overall, VP provides a scalable, efficient, and complementary pretraining paradigm that incorporates native visual knowledge into foundation models, addressing a critical blind spot in current text-centric LLM training.

Key findings

• Visual pretraining (VP) outperforms text pretraining (TP) on scientific reasoning by up to +3.22 points on GPQA (e.g., 76.24 to 79.29 on Qwen 3.5) and +2.10 points on MMLU-Pro with the same document corpus and pretrained model base (Table 1).
• VP achieves these gains using only approximately 25% of the token budget compared to TP (20B visual tokens vs. 80B text tokens), showing higher data and computation efficiency (Figure 2b).
• VP’s advantage grows proportionally with visual-structure density, with largest gains on documents rich in formulas, tables, and layouts, and negligible gains on low-density text-only pages (Figure 2c).
• Cross-modal alignment metrics improve significantly after VP: paired cosine similarity rises from 0.631 to 0.907, centroid separation drops by 1.004, and Linear CKA increases by 0.088, indicating better shared visual-textual representation space without paired supervision (Table 2a).
• VP improves downstream multimodal visual reasoning benchmarks by up to +5.4 points on ChartQAPro and +4.8 points on MathVista using native multimodal backbones, surpassing TP which yields negligible or negative gains (Table 2b).
• VP’s next visual latent prediction loss converges to a slightly lower loss than TP on continued pretraining, and fine-tuning on VP checkpoints reaches better final loss and faster convergence (Figure 2a).
• Attention map comparisons show VP-trained models attending to semantically relevant visual regions (e.g., question constraints and intermediate solution steps) paralleling text-based reasoning attention (Figure 4).
• VP’s performance saturates around 8,192 retained foreground visual tokens per batch, balancing efficiency and accuracy; higher token budgets do not improve results under fixed hyperparameters (Figure 3).

Threat model

n/a; this work focuses on improved foundation model pretraining rather than adversarial or hostile settings. The paper assumes no access to labeled paired visual-text data and no prior OCR outputs, emphasizing unsupervised learning from native visual data. Adversaries or attackers are not modeled.

Methodology — deep read

1. Threat Model & Assumptions: The work targets foundation model pretraining rather than adversarial attack scenarios, focusing on whether visual cues neglected by text-only pretraining can be incorporated effectively. The model assumes no access to explicit paired text-image annotations or OCR supervision, learning from raw visual scientific documents in an unsupervised, autoregressive manner.

2. Data: The main corpus consists of scientific document PDFs containing densely structured visual elements (figures, tables, equations). The same underlying corpus is used for both VP and TP for strict comparison, with TP using OCR/text parsed output (MinerU2.5 parser) yielding ~80B text tokens, and VP using raw rendered page images filtered into ~20B foreground visual tokens representing sparse non-background patches. Evaluation benchmarks include scientific reasoning datasets GPQA, AIME-25, MMLU-Pro, HLE, as well as multimodal visual reasoning sets MMMU-Pro, ChartQAPro, MathVista, and SFE.

3. Architecture / Algorithm: VP uses a frozen vision tower to encode document pages into dense visual latent tokens. Foreground patches are identified using pixel variance and luminance thresholds to remove blank backgrounds, producing a sparse visual token sequence ordered in raster scan order. These visual tokens are linearly projected into the LLM embedding space and concatenated with textual tokens. The model is trained autoregressively to predict the next visual latent token feature via a contrastive InfoNCE loss with in-batch negatives, paralleling language modeling but operating on continuous visual feature space rather than discrete tokens. The final loss mixes standard text cross-entropy loss with weighted visual latent prediction loss, interleaving visual and text sequences during training.

4. Training Regime: Continued pretraining (CPT) is performed on top of base language models such as Qwen 3.5/3 and Llama 3.2/3.1 Vision variants using the XTuner framework. The vision encoder is frozen; only the LLM, visual projection, and visual prediction heads are updated. CPT training uses ~120B tokens for VP (combined text + visual) vs 180B for TP due to compression. Hyperparameters including learning rate and batch size are held constant across experiments to isolate the effect of representation. This is followed by supervised fine-tuning (SFT) with identical data and recipes to evaluate downstream reasoning. Exact seeds and hardware details are not specified.

5. Evaluation Protocol: Models are zero-shot evaluated after SFT using standardized prompts ('think step-by-step') for reasoning benchmarks, with results based on pass@1 or pass@8 metrics averaged over multiple runs. Visual-structure density stratification categorizes evaluation data by visual content richness. Cross-modal alignment is quantified on 100 held-out image-text pairs using centroid separation, cosine similarity, Linear CKA, and mutual k-NN metrics. Qualitative attention maps probe reasoning evidence regions under visual vs text input.

6. Reproducibility: Code, checkpoints, and specific training details are mentioned to be part of supplementary material and external toolkits like XTuner, but no direct public release is indicated. Dataset provenance is scientific PDFs with matched TP/VP processed representations ensuring comparability. The paper includes detailed algorithmic and objective function formulations to facilitate reproduction.

A concrete example: For the Qwen 3.5 model, the same scientific PDF pages are processed either by extracting text tokens (TP) or rendering images with foreground patch selection (VP). VP embeds the visual tokens through the frozen vision encoder into latent vectors projected into the LLM embedding space. Training proceeds by autoregressively predicting the next visual token in the sequence via a contrastive loss, while standard text tokens in the data stream continue to be handled normally. After continued pretraining, the VP model fine-tunes on scientific reasoning tasks, demonstrating improved performance driven by preservation of layout and visual semantics lost by TP's linearized text representation.

Technical innovations

• Introduction of an autoregressive next-visual-latent prediction objective on sparse foreground visual tokens extracted from raw document page images, allowing foundation models to learn native visual structure without OCR or paired supervision.
• A sparse document visual representation pipeline that filters out blank background patches to compress pages into compact ordered sequences of visual latents aligned with spatial layout, reducing token count and improving computational efficiency.
• Joint pretraining framework that interleaves standard text next-token prediction with visual latent prediction using a shared autoregressive transformer backbone, deeply integrating visual signals into the language model's latent space.
• Demonstration that unsupervised visual pretraining on native document images improves language models' scientific reasoning and cross-modal alignment without requiring any image-text pairs or explicit multimodal annotations.

Datasets

• Scientific document corpus — tens of millions of PDF pages — internal source (parsed to text or rendered to image for TP or VP respectively)
• GPQA Diamond — scientific question-answering benchmark — public
• AIME-25 — scientific reasoning dataset with numeric and formula questions — public
• MMLU-Pro — multi-disciplinary knowledge benchmark — public
• HLE — hard multi-step reasoning evaluation — public
• MMMU-Pro, SFE, ChartQAPro, MathVista — multimodal scientific visual reasoning benchmarks — public

Baselines vs proposed

• Text Pretraining Qwen 3.5: GPQA = 76.24 vs VP = 79.29
• Text Pretraining Qwen 3.5: MMLU-Pro = 83.91 vs VP = 85.09
• Text Pretraining Llama 3.2 Vision: GPQA = 30.24 vs VP = 33.08
• Text Pretraining Llama 3.2 Vision: MMLU-Pro = 50.60 vs VP = 51.52
• Text Pretraining Qwen 3.5: ChartQAPro = 56.42 vs VP = 61.80
• Text Pretraining Llama 3.2 Vision: ChartQAPro = 22.23 vs VP = 27.67
• Text Pretraining Qwen 3.5: pass@1 on AIME = 89.58 vs VP = 90.21
• Text Pretraining Qwen 3.5: pass@1 on HLE = 15.70 vs VP = 16.67

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.09657.

Fig 1: Matched text and visual pretraining from the same scientific-document corpus. In the TP

Fig 2: Visual pretraining scales with retained PDF visual tokens and benefits most from structure-heavy

Fig 3 (page 2).

Fig 3: The main VP setting balances visual-token budget and rendering resolution. Left, with the

Fig 5 (page 4).

Fig 6 (page 4).

Fig 7 (page 4).

Fig 8 (page 4).

Limitations

• VP depends on a pretrained language model and continues pretraining with added visual objectives; it is not a standalone visual-only pretraining approach.
• The experiments focus primarily on scientific PDF documents rich in structured visual elements; applicability to broader, unstructured natural images or videos is untested.
• No adversarial robustness or attacks are evaluated, leaving questions about security or manipulations in the visual domain open.
• The visual encoder is frozen during training, potentially limiting end-to-end optimization of visual feature extraction.
• Lack of public release of code and full exact experimental details may hinder reproducibility and broader adoption.
• Optimization dynamics with larger visual token budgets were not fully explored or tuned, perhaps leaving some performance gains untapped.

Open questions / follow-ons

• How can visual-latent decoding be better coordinated with text decoding to maximize joint modeling efficiency and effectiveness?
• Is this visual pretraining paradigm effective on other visually rich but less structured corpora, e.g., natural images, web pages, or video frames?
• Can parameter-efficient modules or adaptive loss scheduling improve VP optimization and scalability with larger visual token budgets?
• To what extent can end-to-end tuning of vision and language components further enhance cross-modal representation learning?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this work underscores the potential of leveraging native visual signals—such as layouts, graphical structures, and equation formats—instead of relying solely on extracted text tokens. CAPTCHAs or bot-detection methods that exploit visual complexity might thus benefit from models pretrained on raw visual documents, which demonstrate enhanced reasoning and multi-modal perception. The findings imply that incorporating visual pretraining can yield more robust and nuanced language understanding in tasks where visual context matters, such as analyzing document screenshots or complex rendered content bots attempt to parse. Moreover, the demonstrated efficiency and scalability suggest the feasibility of deploying such multimodal models in real-world bot detection pipelines that handle diverse visual inputs without extensive OCR or annotation costs.

Cite

@article{arxiv2607_09657,
  title={ Scalable Visual Pretraining for Language Intelligence },
  author={ Yiming Zhang and Zhonghan Zhao and Wenwei Zhang and Haiteng Zhao and Tianyang Lin and Yunhua Zhou and Demin Song and Kuikun Liu and Haochen Ye and Haian Huang and Yuzhe Gu and Haijun Lv and Qipeng Guo and Bin Liu and Gaoang Wang and Kai Chen },
  journal={arXiv preprint arXiv:2607.09657},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.09657}
}

Read the full paper

• arXiv:2607.09657 (abstract)
• arXiv:2607.09657 (PDF)