Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue
Source: arXiv:2606.31719 · Published 2026-06-30 · By Nan Li, Albert Gatt, Massimo Poesio
TL;DR
This paper investigates whether modern vision-language models (VLMs) can accurately distinguish between potential and established common ground in asymmetric collaborative dialogue, using the HCRC MapTask corpus annotated with perspectivist landmark interpretations. The key innovation is framing interpretation matching as a binary classification task predicting whether both dialogue participants share the same landmark interpretation for a given referring expression (RE). They systematically manipulate map information access and dialogue context length to probe model behavior. The authors find that while providing authentic map images or textual map descriptions improves overall model performance, it creates a strong over-alignment bias: models over-predict that participants share common ground when landmarks co-occur on both maps, conflating potential common ground (what could be shared) with what has been pragmatically established through grounding. This bias is absent when the inputs are non-informative visuals, indicating the influence stems from map content rather than visual modality. Further calibration analysis and reference-chain tracking confirm that models rely heavily on static referential cues on the maps and insufficiently track dialogue history or grounding progression. The findings hold most clearly for Qwen3-VL-8B-Instruct and to varying extents across four other VLM variants.
Key findings
- Providing authentic map images improves macro-F1 from 0.591 (text-only) to 0.671 (both maps) at maximum dialogue context (startT) for Qwen3-VL-8B-Instruct (Table 1).
- Map access shifts recall trade-off: recall of aligned (positive) cases rises from 0.590 to 0.822 but recall on non-aligned (negative) cases drops from 0.677 to 0.518, indicating over-prediction of alignment.
- Textual descriptions of map content (landmark names or discrepancy details) reproduce the over-alignment bias with yes-rates between 0.675-0.716 close to authentic maps (0.727) (Table 2).
- Non-informative visual conditions (blank or shuffled maps) suppress alignment predictions completely, resulting in very low yes-rates (~0.18-0.19), showing bias is content-driven not visual modality-driven.
- Calibration analysis shows better calibration on aligned cases (ECEyes = 0.094) but severe miscalibration on non-aligned cases (ECEno = 0.403) with map inputs.
- Accuracy by grounding status reveals that maps improve aligned RE accuracy from 0.590 to ~0.82 but degrade pending (not yet grounded) accuracy from 0.691 to ~0.52, reflecting catastrophic trade-off (Table 4).
- Repeated mentions of the same referent increase yes-rate (over-alignment) under map conditions, inflating perceived accuracy on longer reference chains (Fig 2).
- Cross-model comparisons show the bias arises variably: Qwen3-VL-8B-Instruct, 2B, and Gemma3-4B show over-alignment when maps added; Qwen3-VL-4B shifts more conservative; Gemma3-12B performs very conservatively with low yes-rates, suggesting architecture and visual encoder differences affect behavior.
Threat model
The adversary is modeled as a passive overhearer VLM observing an asymmetric dialogue with access to either text dialogue only, single or both participants’ map views (image or text descriptions), but unable to interact, query, or influence grounding. The model must infer whether the participants have reached shared interpretation despite asymmetry and without grounding actions, reflecting limitations of overhearers lacking interactive grounding capabilities.
Methodology — deep read
The authors formulate the interpretation-matching task as follows: given a MapTask dialogue excerpt with a marked referring expression (RE), the model must predict if the two participants (giver and follower) share the same grounded interpretation. The task is binary: YES if interpretations match, NO otherwise (misunderstandings or pending grounding). The gold label distribution is imbalanced, with 72.1% aligned (YES) and 27.9% not aligned (NO).
Threat model: The model acts as an overhearer with access to either the dialogue transcript, one or both maps (images or textual descriptions), or non-informative visual controls, but cannot interact or query the participants. The adversary is thus an analytical model assessing established common ground from passive observation under asymmetric information conditions.
Data comes from 13,077 perspectivist-annotated REs from 128 dialogues in the HCRC MapTask corpus. Each RE is linked to landmark IDs on giver and follower maps indicating their individual interpretations, enabling precise interpretation matching evaluation.
Context windows vary dialogue history length: curL (current utterance line to target RE), curT (full current transaction), startL (start to target RE line), startT (start through current transaction end). This examines grounding progression effects.
Map access conditions include: no maps (text-only), both authentic maps, giver-only, follower-only maps, plus textual map representations (landmark names, discrepancy descriptions) and non-informative visual controls (blank maps, shuffled landmark maps).
Models evaluated include five open-source VLMs from Qwen3-VL (2B, 4B, 8B-Instruct) and Gemma3 (4B, 12B) families, constrained by a single A100 GPU. The main model is Qwen3-VL-8B-Instruct, selected for best preliminary F1 macro.
Inference uses vLLM with greedy decoding (temperature 0) and constrained output to YES/NO tokens for clean binary decision. Images resized to 1024 pixels max side.
Evaluation metrics: accuracy, macro-averaged F1, recall per class (aligned/non-aligned), yes-rate (proportion YES predicted) as a bias measure, and calibration error (Expected Calibration Error) computed from token logits. Analyses include calibration by class, status-level breakdown by grounding category (aligned/pending/misunderstood), and reference-chain tracking to assess repeated mention effects.
A zero-shot prompting method frames models as overhearer dialogue analysts aware of map asymmetry and task instructions.
An example instance: given a dialogue snippet with the target RE "the parked van," and access to maps or textual descriptions showing two vans on the giver's map but one on the follower's, the model must decide if both participants have grounded their references to the same landmark. The model with map access tends to over-predict YES due to landmark co-presence, even when participants disagree.
Cross-model comparisons probe generality and analyze how different visual encoders and model sizes influence performance and bias manifestation.
Technical innovations
- Formulating interpretation matching in asymmetric dialogue as a binary task explicitly distinguishing potential versus established common ground based on perspectivist annotations.
- Systematic manipulation of map information modalities (authentic images, textual descriptions, non-informative visuals) alongside controlled dialogue context windows to isolate effects on VLM grounding judgments.
- Calibration and reference-chain analyses revealing that VLMs over-weight static referential cues from maps and fail to track incremental linguistic grounding through dialogue history.
- Cross-family model comparison highlighting architectural and visual encoder differences' effects on grounding bias and calibration in vision-language models.
Datasets
- HCRC MapTask perspectivist reference expression annotations — 13,077 REs from 128 dialogues — public (Li et al., 2026)
Baselines vs proposed
- Qwen3-VL-8B-Instruct text-only: F1macro = 0.591 vs both-maps: 0.671
- Recall aligned (pos) text-only: 0.590 vs both-maps: 0.822
- Recall non-aligned (neg) text-only: 0.677 vs both-maps: 0.518
- Qwen3-VL-4B both-maps: F1macro drops to 0.411 from text-only 0.518 (conservative shift)
- Gemma3-12B text-only F1macro: 0.322 vs both-maps: 0.416 (very conservative, low yes-rate)
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.31719.
Fig 1: The interpretation matching task, illustrated on a misalignment example. Panel 1 shows a simplified
Fig 6: Giver’s map for map pair 0 (map0g), used for the spatial-description comparison in Table 8.
Limitations
- Models tested only in overhearer role; no interaction or grounding repair capabilities evaluated, limiting real-world applicability to dialogue agents.
- Evaluation restricted to MapTask corpus, a single task context with hand-drawn maps — may not generalize to other dialogue domains or visual settings.
- Analysis focuses on static model inference without fine-tuning or adaptation to grounding tasks, leaving open potential for improved architectures.
- Calibration and behavioral assessments do not extend to adversarial grounding or explicit misunderstanding scenarios beyond label imbalance.
- Large models with higher parameters do not consistently perform better, indicating unresolved scalability or model architecture issues.
- Textual map descriptions used match content in images but differ in presentation — subtle visual spatial cues may be underexplored.
Open questions / follow-ons
- How can VLM architectures be adapted or trained to better track incremental linguistic grounding over dialogue history rather than relying on static referential cues?
- Would end-to-end training with explicit grounding feedback or interaction components reduce the over-alignment bias and improve calibration on non-aligned instances?
- Can similar evaluation methodologies and interpretation matching tasks be generalized to other multimodal dialogue domains beyond map navigation?
- How can models incorporate uncertainty or represent multiple interpretations to handle persisting misalignments in grounded dialogue contexts?
Why it matters for bot defense
This paper’s findings highlight a critical vulnerability in contemporary vision-language models when used to interpret reference and shared understanding in asymmetric dialogue. For bot-defense and CAPTCHA practitioners, it reveals that VLMs tend to overestimate common ground based on potential referential overlap from visual or textual context, rather than reliable interaction-based grounding. This suggests caution when deploying VLMs as dialogic or interpretive components in bot detection or anti-spam systems: they may confidently misread disagreement or non-alignment as agreement, reducing robustness against spoofing or deception where shared understanding is subtle or incomplete.
Practitioners should consider how grounding failures and over-confidence arise from static scene content cues rather than dynamic interaction histories—designing challenge-response mechanisms or defense CAPTCHAs that require genuine grounding or clarification acts may counteract systematic biases in VLM overhearers. The study also illustrates the importance of finely controlled multimodal evaluation schemes with content versus modality isolation for understanding model behavior, which can inform adversarial testing and robustness analysis in CAPTCHA and bot-defense systems that leverage vision-language understanding.
Cite
@article{arxiv2606_31719,
title={ Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialogue },
author={ Nan Li and Albert Gatt and Massimo Poesio },
journal={arXiv preprint arXiv:2606.31719},
year={ 2026 },
url={https://arxiv.org/abs/2606.31719}
}