CoCo-IR: Contextual Composed Image Retrieval

Source: arXiv:2608.05149 · Published 2026-08-05 · By Shengcao Cao, Tanmaya Shekhar Dabral, Zhongli Ding, Madhuri Shanbhogue, Kaifeng Chen, Zhe Li et al.

TL;DR

This paper addresses the limitation of existing Composed Image Retrieval (CIR) systems which operate only in a single-turn setting, unable to capture the natural, iterative, multi-turn nature of real-world image search interactions. The authors introduce a new task called Contextual Composed Image Retrieval (CoCo-IR), where users refine queries over multiple turns, and retrieval depends on the full history of images and instructions. To tackle this task, they propose a novel Transformable Image Embedding (TIE) model built on a Large Multimodal Model (LMM). TIE uses a specialized embedding token and a hybrid attention mechanism to natively process sequential interactions with interleaved images and text instructions, producing embeddings that evolve contextually across turns.

Because no large-scale dataset exists for this new multi-turn retrieval setting, the authors develop a scalable LMM-powered data engine that generates high-quality multi-turn contextual retrieval data without human annotation. This engine discovers image pairs, synthesizes instructions with self-reflection to filter quality samples, mines hard negatives via LMM-based verification, and assembles multi-turn dialogue chains with rewritten context-aware instructions.

Extensive experiments demonstrate that TIE achieves state-of-the-art results on standard single-turn CIR benchmarks (e.g., 39.4 mAP@5 on CIRCO) and significantly outperforms existing methods on the new multi-turn CoCo-IR benchmark (44.1 Recall@1 on 4-turn dialogues vs 28.2 for best adapted baselines). The results validate that native multi-turn modeling and large-scale contextual data generation enable robust, interactive image retrieval.

Key findings

• Achieved 39.4 mAP@5 on the challenging single-turn CIRCO benchmark, surpassing prior state-of-the-art.
• On the new CoCo-IR multi-turn retrieval benchmark, TIE attains 44.1 Recall@1 on 4-turn dialogues, greatly outperforming existing strong methods adapted from single-turn CIR (28.2 4-turn R@1).
• TIE’s specialized 〈EMB〉token leads to better aggregation of full interaction history, verified by improved retrieval accuracy (Tab. 4).
• Hybrid attention with bidirectional attention within turns and causal attention across turns improves fusion of multimodal inputs and modeling of temporal context.
• The LMM-driven data engine autonomously produces high-quality multi-turn training data, with human verification showing 90.5% overall validity on evaluation sets.
• Using 14× fewer training samples than prior synthetic data methods [37], TIE still achieves superior single-turn CIR performance, demonstrating effective data utilization.
• Baseline single-turn CIR models adapted with heuristic context fusion strategies fail dramatically in multi-turn retrieval, collapsing to near-zero recall at deeper turns (Tab. 2).
• Mining and verifying hard negatives with LMM-based pairwise comparison significantly boosts contrastive training effectiveness.

Threat model

n/a — This is not a security paper and does not define an adversarial threat model. The adversary is not modeled.

Methodology — deep read

The authors first formalize the CoCo-IR task which extends traditional single-turn Composed Image Retrieval into a multi-turn interactive dialogue. At each turn t, the model must retrieve a target image It from a corpus given the entire preceding history Ht (source image plus previous turns of interleaved images and instructions). Success is measured by the strict m-turn Recall@k metric, requiring correct retrieval at every turn to preserve context coherence.

Their proposed Transformable Image Embedding (TIE) model is based on a Large Multimodal Model (LMM) architecture but enhanced for multi-turn embedding generation. Inputs at each turn (images and text instructions) are tokenized and concatenated into a unified token sequence representing the full interaction history. Images are encoded into tokens using an image encoder; text is tokenized conventionally.

A novel special token 〈EMB〉 is appended at the end of each turn’s input sequence. The model outputs the embedding by passing the hidden state of this token through a linear projection layer. This 〈EMB〉token acts as a global bottleneck that aggregates all prior context, improving on prior approaches that used the last token embedding biased towards recent inputs.

The transformer attention is hybrid: within each turn, full bidirectional attention facilitates strong fusion of text and image features; across turns, causal attention enforces temporal flow and prevents information leakage from future turns.

Training uses a contrastive InfoNCE loss. Positive pairs link the query embedding (from the full history and current instruction) with the corresponding target image embedding. The negative set includes all other targets, source images, and model-mined hard negatives in the batch, pushing embeddings apart.

To train at scale, they build a fully automated LMM-powered data engine. It first discovers semantically related image pairs via feature clustering and metadata co-occurrence heuristics. Then an LMM generates transformation instructions and performs self-reflection scoring to select pairs with clean, unambiguous instructions.

Hard negatives are mined by retrieving nearest neighbors in embedding space and using another LMM verifier to ensure they violate at least one aspect of the instruction while still being visually similar.

Multi-turn chains are constructed by linking single-turn image pairs where the target image of one turn becomes the source of the next. The LMM rewrites subsequent instructions to explicitly reference prior context for natural dialogues.

Data is split by image clusters to ensure generalization. Human verification confirms dataset quality.

A detailed experimental protocol tests single-turn CIR benchmarks and the new multi-turn CoCo-IR benchmark. Baseline CIR models are adapted via three strategies to handle multi-turn context, including concatenation, latest inputs only, and LMM-based context summarization. Ablations analyze the importance of the 〈EMB〉token, attention mask design, and hard negative mining.

Training details such as batch sizes, number of epochs, seeds, and hardware were not fully specified in the truncated source. Code and dataset releases were not explicitly mentioned, but project page URL is provided.

Technical innovations

• Introduction of a special 〈EMB〉token as a global information bottleneck to aggregate full multi-turn context into a compact embedding, improving over last-token embeddings.
• A hybrid attention mechanism applying bidirectional attention within turns for deep multimodal fusion and causal attention across turns for temporal sequence modeling.
• Development of a scalable LMM-powered data engine that autonomously generates multi-turn contextual retrieval training data with model self-reflection and LMM-guided hard negative mining.
• Construction of natural, coherent multi-turn retrieval dialogue chains by linking single-turn samples and rewriting instructions with explicit context references.

Datasets

• CoCo-IR Multi-turn Benchmark — thousands of multi-turn dialogue retrieval sequences — Generated by the authors via autonomous LMM data engine, no human annotation.
• Single-turn CIR Benchmarks: Fashion IQ (FIQ) — size unspecified — public benchmark focused on fashion images.
• CIRR — size unspecified — public human-generated natural language instructions on open-domain images.
• CIRCO — large-scale open-domain benchmark — public.

Baselines vs proposed

• MagicLens [37]: Single-turn CIR R@1 on CIRR = ~33.3 vs TIE-12B: 38.7
• MagicLens [37]: Single-turn CIR mAP@5 on CIRCO = 34.1 vs TIE-12B: 39.4
• Baseline single-turn CIR models adapted via concatenation or latest inputs collapse to near-zero 4-turn Recall@1 (e.g., <5%) vs TIE-12B: 44.1 R@1 (Tab. 2)
• Summarized Context adaptation of baselines improves 4-turn R@1 to ~28.2 but still ~16 percentage points below TIE-12B
• Ablation on 〈EMB〉token and hybrid attention yields performance gains of several recall points (Tab. 4).
• Model trained without hard negatives performs worse than with hard negatives by 3 to 5 recall points.

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.05149.

Fig 1: Comparison between conventional single-turn Composed Image Retrieval (CIR)

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 3: Self-reflection during instruction synthesis. As part of our data engine,

Limitations

• The multi-turn benchmark evaluation uses a static fixed instruction and image sequence; does not simulate adaptive user behavior or real-time interaction.
• Hard negative mining is limited to only one iteration and one hard negative per training sample, possibly constraining difficulty and diversity.
• Training details including total compute cost, hyperparameters, and hardware are not fully disclosed in the paper text available.
• The approach depends heavily on large pretrained LMMs for data generation and verification, which may limit accessibility and reproducibility without substantial resources.
• Evaluation focuses on top-k retrieval metrics but does not measure user experience aspects such as latency or robustness against ambiguous instructions.
• The multi-turn datasets are synthetically constructed via LMMs, so their alignment with real-world dialogue complexity and ambiguities, while partially verified by humans, might have residual biases.

Open questions / follow-ons

• How would the model perform in live interactive settings where user instructions evolve dynamically based on retrieved images (adaptive dialogues rather than fixed trajectories)?
• What is the impact of longer multi-turn interactions beyond four turns on model performance and context retention?
• Can the LMM-powered data engine be extended to incorporate noisy, ambiguous or contradictory user instructions to improve robustness?
• How well does the model handle instructions referencing very long-range history or more complex reasoning (e.g., compositional logic) beyond simple visual transformations?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, CoCo-IR advances the state of interactive visual understanding, enabling systems to track and reason about progressive user instructions over multiple turns. This represents a significant step towards more human-like, fluid multimodal interactions where visual contexts evolve. Techniques like the specialized embedding token and hybrid attention may inspire context-aware feature extraction in CAPTCHA schemes.

The autonomous LMM-powered data generation and hard negative mining are of particular interest for building scalable, high-quality datasets reflecting realistic user behaviors without manual labeling, which could improve training of CAPTCHA challenges requiring nuanced visual-textual reasoning. The strict multi-turn recall metric also highlights the importance of sequential context integrity, relevant for multi-step bot-detection challenges where cumulative interaction fidelity matters. However, deploying such complex interactive models in real-time bot defense may pose latency and resource constraints.

Cite

@article{arxiv2608_05149,
  title={ CoCo-IR: Contextual Composed Image Retrieval },
  author={ Shengcao Cao and Tanmaya Shekhar Dabral and Zhongli Ding and Madhuri Shanbhogue and Kaifeng Chen and Zhe Li and Mojtaba Seyedhosseini and Yu-Xiong Wang and Liang-Yan Gui },
  journal={arXiv preprint arXiv:2608.05149},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.05149}
}

Read the full paper

• arXiv:2608.05149 (abstract)
• arXiv:2608.05149 (PDF)