RSICCLLM: A Multimodal Large Language Model for Remote Sensing Image Change Captioning

Source: arXiv:2606.28266 · Published 2026-06-26 · By Yelin Wang, Zijia Song, Shuo Ye, Chuanguang Yang, Miaoyu Wang, Yong Xu et al.

TL;DR

This paper addresses the task of Remote Sensing Image Change Captioning (RSICC), which seeks to generate natural language descriptions of changes between bi-temporal remote sensing images. Existing approaches rely on conventional deep learning models with limited capacity and suffer from scarcity of annotated RSICC data. To overcome these limitations, the authors propose RSICCLLM: the first post-training framework tailored for large vision-language models specifically for RSICC. They introduce a novel data generation paradigm that synthesizes large-scale instruction data (RSICI) by conditioning on binary change masks and automatically generating text descriptions using a strong multimodal LLM. This leads to the creation of a dedicated RSICC benchmark. The main model, based on Qwen2.5-VL-7B, is enhanced with Difference-aware Supervised Fine-tuning that explicitly extracts temporal change representations using central difference convolutions and Hough transforms to focus model attention on geometric-textural differences. Furthermore, Dual-Negative Preference Optimization (DNPO) refines model output quality by generating a preference dataset (RSICP) from complementary negative sample construction strategies. Extensive experiments show RSICCLLM achieves state-of-the-art accuracy on both in-domain and out-of-domain RSICC benchmarks, despite using only 7 billion parameters, outperforming much larger models. Ablations verify the importance of each component and the efficacy of preference learning with quality negatives. Code and datasets are publicly released to advance RSICC research.

Key findings

• RSICCLLM achieves BLEU-4 score of 62.78 on the in-domain RSICI test set, improving over next best Qwen2.5-VL-72B (BLEU-4=2.65) and other baselines by large margins (Table 1).
• On out-of-domain RSICI test set, RSICCLLM attains BLEU-4 of 47.84, outperforming larger models like Qwen2.5-VL-72B (3.06) and Qwen3-VL-235B (2.65) substantially (Table 2).
• Difference-aware Supervised Fine-tuning (SFT) boosts performance from base (BLEU-4=3.21) to 61.68, while adding DNPO further improves BLEU-4 and ROUGE metrics by approx 1.1% and 1.2% respectively (Table 5).
• Ablations show all three SFT components (difference map T, geometry-texture features ΔS, and geometric statistics G) contribute to gains; removing T causes largest drop in BLEU-4 from 62.78 to 60.45 (Table 4).
• Dual-negative preference optimization using info-filtered negative selection and replacement-based negative sample generation each improve BLEU-4 by roughly 0.5%–0.7%, combined gain of 1.1% over SFT alone (Table 6).
• The RSICI instruction dataset contains 40,000 image pairs and 20,000 fine-grained change captions, largest of its kind for RSICC (Table 3).
• RSICP preference dataset includes 20,000 image pairs with positive-negative caption pairs constructed from complementary strategies facilitating preference training.
• The model with only 7B parameters surpasses much larger models (up to 72B parameters) in RSICC, showing task-specific post-training plus high-quality synthetic data effectively overcomes capacity/data limits.

Threat model

n/a — The paper focuses on improving semantic captioning accuracy and interpretability for remote sensing change images rather than explicit adversarial threats or attack-resistant defense.

Methodology — deep read

The paper proposes a three-stage post-training framework RSICCLLM for RSICC:

1. Threat Model & Assumptions: The system assumes a large vision-language model deployed to caption changes between bi-temporal remote sensing images. The adversary model is not explicitly threat-based but RSICC requires fine-grained temporal change understanding robust to irrelevant variations (illumination, viewpoint). The model must sensitivity highlight real semantic changes while ignoring noise.

2. Data: Authors integrate multiple remote sensing change detection datasets: LEVIR-CD, SYSU-CD, S2Looking, and CDD, aggregating ~40,000 image pairs. Each pair has binary change masks indicating change regions. Using Qwen-VL-Max, a strong multimodal LLM, and designed prompts incorporating masks as geometric priors, the authors generate natural language instructions describing precise land surface changes, yielding the Remote Sensing Image Change Instruction (RSICI) dataset. It is split into 32k train, 3k in-domain test, and 5k out-of-domain test samples. Further, the authors create RSICP, a preference dataset with 20k image pairs featuring positive and carefully constructed negative captions for preference learning.

3. Architecture/Algorithm: The backbone model is Qwen2.5-VL-7B, a large multimodal LLM. To enhance change understanding, Difference-aware Supervised Fine-tuning (SFT) explicitly extracts change cues by: (a) performing spatial registration to remove pixel misalignment; (b) computing a preliminary difference map T = I1 - I2 as initial change magnitude cue; (c) applying eight Central Difference Convolution (CDConv)-based kernels to extract geometry-sensitive texture features for each image; (d) applying soft Hough transforms to project convolution responses into parameter space, accumulating geometric evidence; (e) reconstructing spatial geometric features and differencing them to obtain ΔS; and (f) fusing T and ΔS via cross-attention to produce enhanced change-aware visual embeddings. The combination V_enhanced = V_orig + F is fed to the language decoder for captioning.

For post-SFT refinement, Dual-Negative Preference Optimization (DNPO) is introduced. The authors categorize caption words into keywords (object, verb, direction, quantity) and non-keywords. Two negative sample generation strategies are employed:

• Info-filtered negative selection: generates K candidate captions with RSICCLLM-sft, filters to keyword sequences, and selects those with moderate similarity to ground truth as negatives.
• Replacement-based negative construction: randomly replaces keywords in ground-truth captions with semantically similar but incorrect tokens from a contrastive pool to produce hard negatives. These datasets are combined as RSICP for preference training using the Direct Preference Optimization (DPO) loss, plus a KL divergence regularizer for stable training.

4. Training: Models are trained on 2 NVIDIA A100 GPUs (details in Appendix B.2). SFT uses an autoregressive cross-entropy loss over the generated captions conditioned on bi-temporal images plus prompts. DNPO fine-tunes the SFT model with preference loss on RSICP.

5. Evaluation: The authors propose a task-specific RSICC benchmark using RSICI test sets with metrics BLEU-1 to 4, ROUGE-1/2/L, and Sentence-BERT Similarity (SBS) that captures semantic similarity. Additional manual evaluation by GPT-5.2 judges checks object, direction, verb, and magnitude attributes. Baselines include general multimodal LLMs (Qwen, GLM, InternVL) and prior RSICC methods. Ablations isolate contribution of SFT+DNPO and components such as difference map T, geometry features ΔS, geometric statistics G, and negative sample strategies. Training size is varied to test data efficiency.

6. Reproducibility: The paper states code and datasets (RSICI, RSICP) will be publicly released at https://github.com/keaill/RSICCLLM, enabling reproduction. Dataset construction and data generation prompts/algorithms are detailed. Some components such as exact hyperparameters are in the appendix. Large pretrained backbone Qwen2.5-VL-7B is publicly available.

Example Walkthrough: For one bi-temporal image pair, the pipeline performs image alignment, difference map computation, geometric feature extraction via central difference convolutions and Hough voting, then fuses these to enhance visual embeddings. This is combined with textual prompts and fed to the decoder for caption generation. During preference training, the model generates multiple captions; those moderately similar to ground truth are selected as negatives, plus negatives created by keyword replacement. The preference loss pushes the model to rank the ground-truth caption higher, improving descriptive accuracy.

Technical innovations

• A novel instruction data generation paradigm using a multimodal LLM conditioned on bi-temporal images and binary change masks to create a large-scale RSICC-specific instruction dataset (RSICI).
• Difference-aware Supervised Fine-tuning integrating central difference convolution and soft Hough transform features to explicitly extract geometric and texture change representations for enhanced temporal difference perception.
• Dual-Negative Preference Optimization (DNPO) employing two complementary negative caption construction strategies (info-filtered selection and keyword replacement) to form a novel RSICP preference dataset for stable and effective preference training.
• Demonstrating that post-training a moderate-scale 7B parameter large vision-language model with task-specific fine-tuning and preference optimization can outperform much larger models (up to 72B) on RSICC tasks.

Datasets

• RSICI — 40,000 bi-temporal image pairs with 20,000 fine-grained change captions — generated using Qwen-VL-Max with integrated LEVIR-CD, SYSU-CD, S2Looking, and CDD datasets
• RSICP — 20,000 bi-temporal image pairs with 10,000 preference caption pairs — constructed via dual-negative strategies for preference training

Baselines vs proposed

• Qwen2.5-VL-72B: BLEU-4 = 2.65 vs RSICCLLM: 62.78 (in-domain test set)
• Qwen3-VL-235B: BLEU-4 = 2.82 vs RSICCLLM: 62.78 (in-domain test set)
• Qwen2.5-VL-72B: BLEU-4 = 3.06 vs RSICCLLM: 47.84 (out-of-domain test set)
• Base Qwen2.5-VL-7B before SFT: BLEU-4 = 0.61 vs RSICCLLM-sft: 61.68
• RSICCLLM-sft + DNPO: BLEU-4 = 62.78 vs RSICCLLM-sft alone: 61.68
• SFT only BLEU-4 = 61.68 vs SFT + info-filtered negatives BLEU-4 = 62.38 vs SFT + replacement-based negatives BLEU-4 = 62.16

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.28266.

Fig 1: The differences between the previous approaches and ours. (a) Previous meth-

Fig 2: Overview of the proposed RSICCLLM framework. Left: Instruction

Fig 3: Two complementary negative sample construction strategies: (a) semantic de-

Fig 4: Ablation on Difference-aware SFT with different training sizes. Each

Fig 5: Ablation study on whether to employ LS.

Fig 6: Visual analysis of change captioning from different models for qual-

Limitations

• The instruction and preference datasets are synthetically generated via prompting an LLM rather than manually annotated, which may introduce subtle biases or hallucinations despite human curation.
• The evaluation focuses primarily on standard metrics and synthetic plus limited out-of-domain test sets; real-world robustness including sensor noise, atmospheric effects, and adversarial changes remain untested.
• Model relies on spatial registration preprocessing which may be imperfect in complex terrain or large viewpoint shifts, potentially limiting robustness.
• Preference learning depends heavily on negative sampling heuristics which may not cover all error modes; reliance on keyword extraction may miss nuanced semantic errors.
• Methodology centered around one specific backbone (Qwen2.5-VL-7B), and findings may not generalize to other large vision-language models without adaptation.
• Training and inference resource requirements (2x A100 GPUs) may limit application in smaller organizations or real-time processing.

Open questions / follow-ons

• How well does the learned model generalize to remote sensing imagery from different sensors, resolutions, or geographic regions beyond the combined dataset domains?
• What is the robustness of RSICCLLM under natural perturbations such as atmospheric haze, seasonal changes, shadows, or sensor noise?
• Can the preference learning approach be further improved with human-labeled preference data or reinforced with adversarial robustness objectives?
• How adaptable is the difference-aware fine-tuning paradigm to other temporal-language tasks such as video captioning or continuous monitoring?

Why it matters for bot defense

This research is highly relevant to CAPTCHA and bot-defense engineers interested in complex, multimodal verification tasks involving temporal image understanding. The approach demonstrates how large-scale vision-language models can be post-trained to recognize subtle changes between bi-temporal images and generate precise semantic descriptions, moving beyond simpler binary detection or segmentation tasks.

For CAPTCHA design, the Difference-aware Supervised Fine-tuning technique provides a methodology to improve model sensitivity to relevant temporal differences while ignoring irrelevant noise, which could inspire more advanced challenge generation or verification involving temporal visual proof-of-work. Similarly, the dual-negative preference optimization mechanism offers a strategy to refine large models toward nuanced semantic distinctions critical for robust human verification. While remote sensing is a distinct domain, the core techniques of instructional data generation, difference embedding, and preference refinement offer transferable insights for advancing CAPTCHA systems reliant on temporal scene verification or change detection.

Cite

@article{arxiv2606_28266,
  title={ RSICCLLM: A Multimodal Large Language Model for Remote Sensing Image Change Captioning },
  author={ Yelin Wang and Zijia Song and Shuo Ye and Chuanguang Yang and Miaoyu Wang and Yong Xu and Zhulin An and Yongjun Xu and Zitong Yu },
  journal={arXiv preprint arXiv:2606.28266},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.28266}
}

Read the full paper

• arXiv:2606.28266 (abstract)
• arXiv:2606.28266 (PDF)