VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation

Source: arXiv:2607.28590 · Published 2026-07-30 · By Kangning Zhang, Yixing Li, Shuai Shao, Qingyao Li, Zhengxi Lu, Zhiyuan Yao et al.

TL;DR

The paper addresses a key limitation in multimodal on-policy distillation (OPD) for visual language models: the teacher's next-token corrections are source-mixed, blending signals from actual visual evidence with linguistic priors and teacher idiosyncrasies. This confounds effective knowledge transfer since some teacher corrections lack clear visual grounding. The authors propose Visual Attribution Distillation (VAD), a novel counterfactual target-reconstruction method that isolates the component of the teacher's correction truly attributable to visual evidence. VAD estimates a signed proxy vector representing the visual evidence direction by comparing teacher outputs on evidence-present and evidence-removed image views for a fixed student-generated prefix. It then projects the original teacher correction onto this proxy to obtain a visually aligned correction component, reconstructing a student-anchored training target from this rather than relying on the full mixed teacher target. This disentangles visual and non-visual components to supply a cleaner visual supervision signal with a weak privileged-teacher regularizer. Experiments across six fine-grained visual benchmarks on two Qwen3.5 student model scales (4B and 9B parameters) show that VAD consistently outperforms prior OPD methods including direct privileged-view distillation, visual advantage weighting, and decomposed OPD by 2-3 accuracy points on average. Semantic analyses confirm that the visually attributed correction enriches for visually relevant and task-critical tokens, effectively supporting correct answers and stronger refutation of mistaken ones. VAD thus offers a principled alternative to source-mixed supervision for improved fine-grained visual knowledge transfer during multimodal distillation.

Key findings

• VAD improves fine-grained visual accuracy on six benchmarks over scale-matched baselines by 2.40 points (4B) and 2.80 points (9B) in average accuracy (Table 1).
• The visually attributed correction component rvis_t concentrates 42.0% of top-5 token corrections linked to visual attributes, objects, and answer decisions vs 26.7% for original correction and only 17.7% for residual (Figure 3a).
• VAD’s proxy-based projection isolates signed visual support and refutation, increasing correct-token support by up to 7.89 percentage points and wrong-token suppression by 6.61 points in offline evaluation versus full teacher correction (Figure 4).
• Branch-separated target reconstruction (qVAD_T) provides a 1.00 point Avg6 accuracy gain over a one-sided projection target without branch separation (Table 2).
• Inclusion of a weak privileged-teacher regularizer avoids unstable output behavior and yields an additional 0.80 point accuracy improvement (Table 2).
• VAD outperforms direct privileged-view distillation and multiple prior OPD variants consistently on all six benchmarks across both model scales, highlighting broad improvements.
• The proxy unexplained residual rres_t captures mainly linguistic, formatting, and teacher-specific effects distinct from visual corrections (Figure 3).
• The visual intervention proxy vector ut captures token-level changes in centered log-probabilities between evidence-present and evidence-removed teacher distributions, effectively serving as a signed proxy for visual evidence direction.

Methodology — deep read

Threat Model & Assumptions: The adversary context is not explicitly defined as this is a distillation-focused paper rather than a traditional security threat study. The setup assumes a fixed teacher model, stronger than the student, with privileged access to controlled visual evidence (region crops or degraded views). The main assumption is that the teacher's corrections can be decomposed into parts attributable or not attributable to visual evidence changes via counterfactual interventions.

Data: Training data consists of 6,241 synthetic visual question-answering examples derived from Vision-OPD releases. Each example includes a full image (x0), an evidence-present crop (x+), and an evidence-degraded crop (😆, constructed by downsampling the crop and then upsampling back to the original size, preserving region and size but degrading fine-grained evidence. The datasets for evaluation include six fine-grained visual benchmarks: VStar, ZoomBench, HRBench (4K and 8K), and MME-RealWorld (EN and CN).

Architecture / Algorithm: VAD trains student models initialized from Qwen3.5-4B and Qwen3.5-9B checkpoints. At each student-generated token prefix y<t, three next-token distributions are evaluated: the student on full image p0_S, and the fixed teacher on evidence-present crop p+_T and evidence-removed crop p-_T under the same prefix. These distributions are restricted to the top-K student candidate tokens per position, mapped to centered log-probability vectors ϕ_t(p).

The privileged teacher correction rt is defined as the difference in centered logits: rt = ϕ_t(p+_T) - ϕ_t(p0_S), representing the complete correction to the student relative to full image inputs. The intervention-derived proxy vector ut = ϕ_t(p+_T) - ϕ_t(p-_T) encodes the signed visual evidence direction at this token position (positive values support, negative values refute tokens).

VAD projects rt onto ut to obtain the visually aligned component rvis_t = β_t ut where β_t = max(inner(rt, ut),0) / (||ut||^2 + ζ), clipping negative alignments to zero to keep a one-sided projection. This isolates correction directions consistent with the visual intervention.

A refined, budgeted support/refutation decomposition splits ut into positive u+_t and negative u-_t branches, separately allocating the visual-correction budget B_t = ||rvis_t||^2 based on agreement scores with rt. This produces a final refined correction rVAD_t aggregating support/refutation components.

The student-anchored reconstructed supervision target qVAD_T,t = softmax(ϕ_t(p0_S) + clip(rVAD_t, -c, c)) modifies only the student's current distribution by shifts attributable to visual evidence rather than the whole privileged teacher distribution.

Training Regime: Training uses a token-level Jensen-Shannon divergence loss D_JS(qVAD_T,t, p0_S) averaged over valid response positions T, with an additional weak regularizer L_reg from the original privileged teacher distribution p+_T scaled by the fraction of unattributed correction (1 - ρ_t). Loss L = L_vis + λ L_reg, with λ=0.1.

Batch size is 96, with 8 rollouts per prompt, learning rate 2e-6. The stabilizing projection parameter ζ=1e-3 and max logit shift clip c=20. Positive branch cap τ+ is 0.8 for 4B and 0.7 for 9B models.

Evaluation Protocol: Evaluation uses official Vision-OPD inference pipeline with GPT-OSS-120B as a judge model for six fine-grained visual benchmarks. Accuracy (%) aggregates across benchmarks reported as Avg6 mean. Baselines include Vision-OPD, VA-OPD, V-Zero, GRPO, Decomposed OPD under matched data and update budgets. Ablations examine target construction variants, inclusion of regularizers, and alternative projections. Controlled offline analyses assess token-level probability shifts on correct and mistaken tokens. Semantic analyses cluster top-corrected tokens by category to interpret attribution.

Reproducibility: Code and models are released publicly on GitHub and HuggingFace (links provided). Training uses frozen teacher weights and synthetic datasets from Vision-OPD. All teacher-generated intermediate views (evidence present vs removed) are required only during training; inference runs the student on full images only. Hyperparameters and algorithmic details are thoroughly documented.

End-to-end example: At a given token position t along a student-generated prefix y<t, the student distribution p0_S and teacher distributions p+_T, p-_T are computed on full image, crop (evidence present), and crop (evidence removed). Centered log probabilities ϕ_t are obtained, rt and ut computed, and rvis_t extracted by projection. The final qVAD_T,t target shifts p0_S by the attributed correction, supporting tokens favored by the visual evidence and suppressing those refuted. The student receives cross-entropy supervision on qVAD_T,t. This process repeats for all tokens and batches, with weak regularization anchored on privileged teacher full correction p+_T.

Technical innovations

• Introduction of a counterfactual visual attribution proxy vector ut that quantifies signed token-level support or refutation due to a controlled visual intervention, enabling decomposition of teacher corrections.
• A one-sided projection of the privileged teacher correction rt onto the visual proxy ut to isolate visually attributable components, separating visual corrections from linguistic or teacher-specific residuals.
• Reconstruction of student-anchored training targets qVAD_T,t by applying the visually attributed correction onto the student's current token distribution rather than matching the full privileged teacher distribution.
• Separate budget allocation to support and refutation branches of the visual proxy, enabling nuanced correction shifts that both raise correct tokens and suppress incorrect ones in a fine-grained manner.

Datasets

• Vision-OPD synthetic training set — 6,241 examples — public release from Yuan et al. (2026)
• VStar — size unspecified — fine-grained visual benchmark
• ZoomBench — size unspecified — fine-grained visual benchmark
• HRBench 4K — size unspecified — high-resolution visual benchmark
• HRBench 8K — size unspecified — high-resolution visual benchmark
• MME-RealWorld EN — size unspecified — real-world recognition benchmark
• MME-RealWorld CN — size unspecified — real-world recognition benchmark

Baselines vs proposed

• Direct privileged-view distillation: Avg6 accuracy = 75.92% (4B) vs VAD 78.32%
• Scalar-shrunk correction supervision: 76.19% vs VAD 78.32% (4B)
• One-sided projection without regularization: 77.06% vs VAD 78.32% (4B)
• One-sided projection with regularization: 77.52% vs VAD 78.32% (4B)
• Decomposed OPD: 75.37% vs VAD 78.32% (4B), and 77.05% vs 79.93% (9B)
• VA-OPD: 74.92% vs VAD 78.32% (4B), 76.67% vs 79.93% (9B)
• Vision-OPD: 75.92% vs VAD 78.32% (4B), 76.88% vs 79.93% (9B)
• Qwen3.5 base model (no OPD): 70.85% vs VAD 78.32% (4B), 73.86% vs 79.93% (9B)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.28590.

Fig 1: Motivation and overview. (A) A privileged-view teacher proposes a correction at every student-generated position,

Fig 2: Overview of VAD. (a) On-policy counterfactual views. Given the same student-generated prefix, the full-image

Fig 3: Semantic separation of the teacher correction. (a) Relative top-5 composition of the complete, visual, and residual

Limitations

• Evaluation datasets and models rely on synthetic data with known crops and degraded images; real-world evidence manipulations may be less controlled.
• Visual intervention proxy ut is an approximate signal, not an oracle causal decomposition; it may miss some subtle or non-token-level visual contributions.
• The approach depends on availability of controlled paired views (evidence-present and degraded) during training, increasing data preparation complexity.
• The weak privileged-teacher regularizer is necessary to avoid output drift but may complicate optimization and hyperparameter tuning.
• Disentangled residual rres_t is not strictly non-visual; interpretation as non-visual remains nuanced and may still contain some visual effects.
• Inference efficiency is improved over multi-view evaluation, but training includes overhead from triple-distribution evaluations and projections.

Open questions / follow-ons

• Can the visual attribution proxy be extended to handle continuous or multi-modal evidence changes beyond binary present/removed crops?
• How robust is VAD to noisier or less perfectly aligned evidence degradations especially on real-world datasets?
• Could the proxy-based attribution approach be adapted to other modalities or tasks with privileged teacher components beyond vision?
• What are the theoretical guarantees or bounds on attribution accuracy with this signed projection approach, and could more principled causal inference methods improve it?

Why it matters for bot defense

For bot-defense engineers working on CAPTCHA or related bot-detection technologies leveraging multimodal models, VAD offers a method to more reliably interpret which teacher corrections are genuinely supported by visual evidence rather than confounded by language priors or model idiosyncrasies. This can lead to more accurate visual grounding in OCR, image recognition, or challenge-response tasks that rely on fine-grained visual understanding. By reconstructing distilled targets focused on verifiable visual cues rather than mixed signals, VAD may improve robustness to adversarial examples or spoofing attempts that exploit linguistic shortcuts. Further, its counterfactual intervention approach provides a conceptual framework for evaluating model sensitivity to manipulated visual inputs, valuable for designing attack-resistant defense layers in CAPTCHA pipelines. However, the reliance on controlled visual evidence pairs limits direct applicability to environments without such privileged data, so adaptation or proxy generation will be critical for deployment.

Cite

@article{arxiv2607_28590,
  title={ VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation },
  author={ Kangning Zhang and Yixing Li and Shuai Shao and Qingyao Li and Zhengxi Lu and Zhiyuan Yao and Jianghao Lin and Wenxiang Jiao and Yuan Lu and Weiwen Liu and Weinan Zhang and Yong Yu },
  journal={arXiv preprint arXiv:2607.28590},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.28590}
}

Read the full paper

• arXiv:2607.28590 (abstract)
• arXiv:2607.28590 (PDF)