OPD-V: Visual On-Policy Self-Distillation with Modality Balance

Source: arXiv:2608.05131 · Published 2026-08-05 · By Aniri, Jinhe Bi, Peng Liao, Zengjie Jin, Volker Tresp, Fei Shen et al.

TL;DR

This paper addresses a fundamental challenge in multimodal large language model (MLLM) reasoning called Modality Imbalance, where textual context tends to dominate over visual input during generation. Existing On-Policy Self-Distillation (OPSD) methods improve MLLM reasoning by leveraging privileged information from different input sources, but they overlook this imbalance, limiting their effectiveness. To tackle this, the authors propose treating the relative balance between visual and textual modalities as privileged information itself. They introduce OPD-V, a visual OPSD framework that uses a dual-teacher design: a Positive Teacher receiving a Zoom-In Image that emphasizes task-relevant visual regions and a Negative Teacher receiving a Mask Image with occluded regions. By comparing these teachers’ token-level logits, they define a Modality-Balance Logits Margin that selects a trust region of tokens for on-policy self-distillation using Jensen–Shannon divergence. Extensive experiments over six challenging multimodal benchmarks, four MLLM backbones, and five post-training methods show that OPD-V consistently outperforms existing OPSD variants and strong baselines, improving average reasoning accuracy by up to 15.7 percentage points on Qwen3.5-4B. Additionally, OPD-V reduces training step latency by up to 31.8% due to shorter generated responses and more efficient privileged input construction. The study demonstrates that explicitly incorporating modality balance as privileged information effectively mitigates the modality dominance problem and leads to more accurate and efficient visual reasoning in MLLMs.

Key findings

• OPD-V raises average accuracy on Qwen3.5-4B from 64.30% to 80.01%, an absolute improvement of 15.71 points across six benchmarks.
• OPD-V outperforms strongest prior OPSD baseline Vision-OPD (77.10%), surpassing it by 2.91 points on average.
• OPD-V reduces mean training step latency by 31.8% on 4B model (352s to 240s) and 24.7% on 9B model (451s to 340s) compared to standard OPSD.
• Average response length during training drops by 74.5% under OPD-V (141 tokens) relative to OPSD (554 tokens), reducing token-level computation.
• The Modality-Balance Trust Region includes roughly 50% of on-policy tokens across training and model scales, maintaining a stable selection signal.
• Using the Positive Teacher alone improves base model accuracy by 10.32 points (74.62% vs 64.30%), and Negative Teacher alone improves by 7.68 points (71.98%), but combining both yields the highest gain (80.01%).
• Replacing the Zoom-In Image or Mask Image in teacher pairs with alternatives like Repeat, Blur, Prune, or No Image reduces accuracy by 4–8 points, confirming the chosen transformations’ efficacy.
• Policy entropy for OPD-V remains stable after warm-up with final rolling means around 0.827 (4B) and 0.761 (9B), showing non-degenerate, active learning.

Methodology — deep read

The authors address the limitation of modality imbalance in MLLMs during on-policy self-distillation (OPSD), where textual information tends to overshadow visual input, weakening distillation effectiveness. They propose to use Modality Balance—the relative visual vs textual attention ratio during token generation—as privileged information. To operationalize this, they construct two teacher models from the same base MLLM architecture (exponential moving average of student parameters):

1. Positive Teacher receives a Zoom-In Image cropped and magnified around task-relevant regions.
2. Negative Teacher receives a Mask Image generated by randomly masking a portion of the zoomed image.

The student receives the Original Image.

During training for each on-policy generated token, all three models score the token logits given identical textual queries and prefixes. The token-level Modality-Balance Logits Margin is computed as the difference in log probabilities assigned by the Positive and Negative Teachers for the student-generated token. Tokens with positive margin form the Modality-Balance Trust Region, representing positions where the positive teacher provides stronger support than the negative teacher.

Self-distillation uses Jensen–Shannon divergence to align the student distribution with that of the Positive Teacher only within this trust region. The margin values weight the distillation loss to emphasize high-confidence tokens. The objective encourages the student to focus on integrating information from visual inputs where modality balance favors visual attentiveness.

The training dataset consists of 6.2K curated visual reasoning samples from Vision-OPD. Training uses a top-K distillation strategy (top 100 logits) to reduce memory overhead, with the teacher as an EMA of student parameters. The maximum on-policy generation length is 1024 tokens. Models are trained for 1 epoch.

Evaluation covers six benchmarks spanning high-resolution and fine-grained reasoning tasks (V* Bench, ZoomBench, HR-Bench at 4K and 8K, and English/Chinese subsets of MME-RealWorld). Four MLLM backbones are tested (Qwen3.5 4B and 9B, Qwen3-VL 4B and 8B).

Baselines include multiple post-training methods (SFT, GRPO, standard OPSD, Vision-OPD, VA-OPD) all reimplemented under matched conditions on the same backbone and data to ensure fair comparison.

Evaluation metrics are accuracy on multimodal reasoning benchmarks and training efficiency metrics including response length, step latency, and policy entropy. Ablations examine the independent contribution of Positive and Negative Teachers, as well as substitute transformations like Repeat and Blur images.

Reproducibility details: the teacher is an EMA of the student, single-epoch training on a public Vision-OPD dataset subset, with distilled logits top-K approximations. Code release is noted but not detailed. Exact seeds or hardware not specified. The methodology section is exhaustive in explaining the mechanism from privileged input construction through token selection to loss computation, supporting end-to-end understanding.

Technical innovations

• Formulating Modality Balance—the internal visual vs textual attention ratio—as explicit privileged information for OPSD in MLLMs.
• Introducing a dual-teacher framework with a Positive Teacher (Zoom-In Image) and Negative Teacher (Mask Image) to define token-level Modality-Balance Logits Margins.
• Defining a Modality-Balance Trust Region that selects tokens with positive margin for focused on-policy Jensen–Shannon distillation.
• Using tokenwise margin weighting to scale distillation loss, emphasizing tokens with stronger visual evidence influence.
• Employing top-K logit distillation and efficient privileged inputs to reduce computational overhead while maintaining distillation quality.

Datasets

• Vision-OPD subset — 6.2K visual reasoning samples — curated synthetic dataset from prior work [4]

Baselines vs proposed

• Qwen3.5-4B Base model: average accuracy = 64.30% vs OPD-V: 80.01%
• SFT (Supervised Fine-Tuning): 66.85% vs OPD-V: 80.01%
• GRPO (RL method): 69.65% vs OPD-V: 80.01%
• OPSD (standard on-policy self-distillation): 69.26% vs OPD-V: 80.01%
• Vision-OPD: 77.10% vs OPD-V: 80.01%
• VA-OPD: 71.88% vs OPD-V: 80.01%
• Positive Teacher only: 74.62% vs OPD-V dual teachers: 80.01%
• Negative Teacher only: 71.98% vs OPD-V dual teachers: 80.01%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.05131.

Fig 1: Modality Balance as privileged information, evaluated on 5K samples drawn from multiple domains. Left:

Fig 2: Overview of OPD-V. (A) For the same student-generated response, the Positive Teacher, student, and

Fig 3: Training dynamics on Qwen3.5 backbones. Thin traces show logged values, and bold curves show seven-step

Fig 7: Training example for a rear-fender reflector question.

Limitations

• Experiments use only one main training dataset (Vision-OPD subset with 6.2K samples), limiting evaluation of OPD-V generalization to other data distributions.
• The method assumes availability of a task-relevant visual crop (Zoom-In Image); automatic or unsupervised determination of this region is not addressed.
• The approach is tested on relatively recent Qwen-based backbones; applicability to other architectures or modalities is unclear.
• No adversarial or robustness evaluation to modality shifts or noisy inputs is reported.
• Training is limited to a single epoch; long-term stability and convergence behavior under OPD-V remain unexamined.
• No explicit ablation studies testing sensitivity to hyperparameters like the margin threshold or top-K value are provided.

Open questions / follow-ons

• Can OPD-V’s modality balance approach be extended to unsupervised or self-supervised fine-tuning where verified targets are unavailable?
• How well does the Modality-Balance Trust Region generalize to other multimodal tasks with more than two modalities or complex input structures?
• What automated methods can reliably and efficiently identify task-relevant visual regions to construct Zoom-In Images without manual annotation?
• How robust is OPD-V to noisy or adversarial perturbations in either the zoom-in or mask images, and how does this affect modality balance?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, OPD-V highlights a promising approach to improving multimodal model performance by explicitly addressing how models balance and integrate multiple input modalities. The Modality-Balance Trust Region concept may inform ways to detect whether a model genuinely uses visual information rather than over-relying on textual context or shortcuts, a known challenge for CAPTCHAs that verify human visual understanding. OPD-V’s dual-teacher design and token-level margin criterion create a finer-grained supervision signal that can enhance model robustness and reasoning accuracy, potentially useful for designing multi-step or interactive challenges combining visual and textual clues. Furthermore, the efficiency gains from shorter outputs and more targeted self-distillation could enable more practical deployment of multimodal reasoning models in resource-constrained or latency-sensitive CAPTCHA systems. However, deployment would require adapting OPD-V’s modality balance measures to specialized CAPTCHA tasks and ensuring automated determination of relevant visual input regions, which are not addressed here.

Cite

@article{arxiv2608_05131,
  title={ OPD-V: Visual On-Policy Self-Distillation with Modality Balance },
  author={ Aniri and Jinhe Bi and Peng Liao and Zengjie Jin and Volker Tresp and Fei Shen and Yunpu Ma and Tat-Seng Chua },
  journal={arXiv preprint arXiv:2608.05131},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.05131}
}

Read the full paper

• arXiv:2608.05131 (abstract)
• arXiv:2608.05131 (PDF)