Unpaired Modality-Agnostic Generative Recommendation

Source: arXiv:2608.02477 · Published 2026-08-03 · By Weihao Shen, Wei Chen, Fuwei Zhang, Meng Yuan, Yuqin Lan, Guojun Liu et al.

TL;DR

This paper tackles a fundamental limitation in multimodal Generative Recommendation (GR) systems, which traditionally require paired image-text data for training semantic ID tokenizers. Such paired data is scarce and hard to maintain in real-world catalogs where images and textual metadata often originate from independent, asynchronous pipelines, resulting in incomplete modality overlap. Existing approaches thus underutilize abundant unpaired visual or textual data, limiting recommendation quality and robustness.

The authors propose Unpaired Modality-Agnostic Generative Recommendation (UnpairGR), a novel framework that learns a unified semantic-ID space from both paired and unpaired observations, including image-only and text-only items. The key insight is to restrict modality-specific processing to lightweight input projections and share the subsequent Transformer model and residual vector quantization codebooks across all modalities and observation conditions. Paired data is used to establish a confidence-weighted semantic consensus, while unpaired data directly refines representations within the same discrete codebook space. This results in a tokenizer producing compatible semantic IDs for all data, enabling a single autoregressive generative recommender without fallback or feature imputation mechanisms.

Extensive experiments on three public recommendation benchmarks demonstrate that UnpairGR consistently outperforms state-of-the-art multimodal GR models under both fully observed and incomplete modality conditions. The learned semantic IDs from different modalities show strong prefix agreement and low discrepancy, confirming the effectiveness of the shared discrete semantic space. Ablation studies verify that the shared Transformer and residual codebooks as well as unpaired supervision are all critical for performance. Overall, UnpairGR significantly pushes the boundary of multimodal GR by effectively exploiting unpaired data and improving robustness and accuracy in heterogeneous, noisy real-world scenarios.

Key findings

• UnpairGR improves top-1 hit rate (HR@1) by 22.86% on Arts and 17.14% on Games compared to the strongest baseline SynGR under fully paired multimodal input (Table 1).
• Under modality missingness conditions (25%-100% missing text or image), UnpairGR maintains the highest HR@10 performance across all three datasets versus strong multimodal baselines MACRec and MQL4GRec (Table 2).
• Cross-observation semantic ID consistency shows UnpairGR yields high prefix agreement (P1 and P2 > 0.85) and substantially lower normalized Hamming distance for text-only and image-only IDs against paired references (Table 3), indicating robust shared discrete semantic space.
• Ablations demonstrate removing the shared Transformer trunk reduces macro-average HR@10 from 0.1477 to 0.1170 and NDCG@10 from 0.1040 to 0.0813; excluding shared residual codebooks or unpaired supervision also causes significant drops (Figure 4).
• Single-modality cold start experiments reveal semantic IDs from unimodal observations (image-only or text-only) support effective recommendation with consistent HR@10 and NDCG@10 gains compared to fully paired routes, enabling cold-start recommendation without feature imputation (Figure 3).
• The shared tokenizer trained on mixed paired and unpaired data enables a single generative recommender to handle heterogeneous item observations, avoiding the need for modality-specific models or fallback mappings.
• Using entropy-based modality reliability weights in the shared quantized space avoids naive averaging and preserves semantic consensus better in paired representations.
• UnpairGR matches or exceeds prior SOTA generative recommendation model accuracy across three public datasets—Arts, Games, and Instruments—confirming generality.

Threat model

The work does not explicitly define an adversarial threat model; rather, it addresses the practical challenge of incomplete or unpaired multimodal item observations in large-scale recommendation catalogs. The implicit assumption is a non-adversarial setting where multimodal inputs are noisy or missing but not maliciously manipulated. The adversary cannot alter the fundamental recommendation model or semantic tokenizer after training.

Methodology — deep read

1. Threat Model & Assumptions: The focus is on item representation learning for next-item recommendation. The adversary model is not explicitly defined as the work is algorithmic rather than adversarial, but the key challenge addressed is the technical difficulty arising from missing or unpaired modality data across items in real-world catalogs. The assumption is that paired, image-only, and text-only item data are available, but item-level image-text pairing is incomplete or inconsistent.

2. Data: Three benchmark multimodal recommendation datasets are used—Arts, Games, and Instruments—with training, validation, and test splits typical to prior works cited. The data includes paired items having both an image and a text description, as well as unpaired image-only or text-only items representing incomplete observations. Data is balanced during training to prevent domination by a particular modality observation type.

3. Architecture / Algorithm: The core architecture consists of frozen modality-specific encoders extracting visual (hv) and textual (ht) features per item. These features pass through lightweight modality-specific linear projections Pm to unify their dimension and statistics, producing zm. A shared Transformer encoder Φθ then maps zm to a modality-agnostic representation rm.

A novel residual vector quantization module with L levels and shared codebooks {C^(ℓ)} across modalities is applied to rm to obtain discrete semantic ID sequences s, where each level selects a codeword minimizing quantization residual. Cross-modal consensus for paired items fuses visual and textual rm with entropy-based reliability weights derived from codebook soft assignment confidence (lower entropy = higher confidence). Unpaired items use their unimodal rm directly.

Semantic ID learning optimizes a mixed-supervision objective combining reconstruction losses (via modality-specific reconstruction heads) for available modalities, plus cross-modal alignment loss minimizing L2 distance before quantization, Jensen-Shannon divergence consistency on code assignment distributions after quantization, and a code usage regularizer encouraging uniform codebook utilization.

Training is staged: first the tokenizer network (projections, Transformer, codebooks) is trained and then frozen. Next, the generative recommender (autoregressive Transformer) is trained using fixed semantic IDs for user histories and next-item targets.

4. Training Regime: The tokenizer is optimized using the mixed objective over balanced mini-batches from paired, image-only, and text-only datasets. Hyperparameters λ, η, ξ control the balance of alignment, consistency, and code usage losses, respectively. Assignment sharpness τ controls soft assignment entropy calculation. The optimizer and exact training details are not fully described, but standard practices presumably apply. Seeds and hardware are not specified.

5. Evaluation Protocol: Recommendation accuracy metrics HR@k and NDCG@k are reported under fully observed and modality missingness conditions. Cross-observation semantic ID consistency is evaluated by exact agreement, prefix agreement, and normalized Hamming distance between semantic ID sequences from paired vs unimodal observations. Ablation studies remove shared Transformer, shared codebooks, or unpaired supervision to quantify component contributions. Cold-start is tested by masking one modality in targets at different rates. Efficiency and convergence are analyzed but details truncated in excerpt.

6. Reproducibility: The authors release code at https://github.com/Nevaeh7/UnpairGR. Dataset details and hyperparameter selections are in supplementary material. Tokenizer weights are frozen in second training stage. The tokenizer merges heterogeneous observations into one discrete semantic space, enabling reproducible semantic IDs from unpaired data. Overall, the methodology is clearly described, but full implementation and reproducibility details require consulting the released code and supplement.

Concrete Example: An item observed both via an image-only modality and paired image-text modalities is processed as follows: image encoder outputs hv, mapped via lightweight projection Pv, then through shared Transformer Φθ producing rv. For the paired modality, textual encoder output ht is similarly projected and transformed to rt. The two are fused via reliability weights derived from codebook assignment entropies to get a consensus rp. Residual vector quantization applies shared codebooks across L layers to rp, producing semantic ID s. For image-only observations, rv alone is quantized with the same shared quantizer to yield compatible semantic ID sequences. These discrete IDs form tokens in the autoregressive recommendation model input sequence for user interaction histories and targets.

Technical innovations

• Unified semantic-ID tokenizer training combining paired, image-only, and text-only observations into a single discrete semantic space using shared Transformer and residual codebooks.
• Entropy-based modality reliability weighting in the shared quantized space to form cross-modal semantic consensus rather than naive averaging.
• Mixed-supervision quantization loss integrating reconstruction, alignment, cross-observation consistency, and code usage regularization to balance diverse modality availability.
• Avoidance of modality-specific codebooks, feature imputation, or fallback models by enabling modality-agnostic semantic ID generation compatible across all observation conditions.

Datasets

• Arts — size not specified — public multimodal recommendation benchmark
• Games — size not specified — public multimodal recommendation benchmark
• Instruments — size not specified — public multimodal recommendation benchmark

Baselines vs proposed

• SynGR: HR@1 = 0.0713 (Arts) vs UnpairGR: 0.0876 (+22.86%)
• SynGR: HR@1 = 0.0245 (Games) vs UnpairGR: 0.0287 (+17.14%)
• SynGR: NDCG@5 = 0.1045 (Instruments) vs UnpairGR: 0.1214 (+16.17%)
• MQL4GRec: HR@10 missing 50% text = 0.1086 (Arts) vs UnpairGR: 0.1202 (+6.09%)
• SynGR: HR@10 image-only missing 25% = 0.0511 (Arts) vs UnpairGR: 0.1169 (+13.83%)
• SynGR: HR@10 random single-modality missing 75% = 0.0675 (Arts) vs UnpairGR: 0.1202 (+6.12%)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.02477.

Fig 1: Existing multimodal GR methods depend on paired

Fig 2 (page 1).

Fig 2: Overview of UnpairGR. Paired and unpaired data update the same tokenizer through thin modality-specific projections,

Fig 4 (page 4).

Fig 3: Cold-start mechanism analysis for UnpairGR.

Fig 5: compares UnpairGR with MACRec and SynGR

Fig 4: Core component ablation study on Arts, Games,

Fig 8 (page 7).

Limitations

• Exact dataset sizes, modality coverage statistics, and full data splits are not detailed in the main text, limiting reproducibility evaluation.
• The work does not explicitly evaluate robustness under adversarial or malicious manipulations of modalities.
• Training configuration such as optimizer choices, learning rates, epochs, and computational resource requirements are not fully disclosed.
• No explicit evaluation of the approach on user-side cold-start scenarios with new users rather than new items.
• Cross-dataset generalization outside the three benchmarks is untested, so transferability to other domains remains uncertain.
• Handling of noisy or low-quality modalities beyond 'missingness' is not empirically analyzed.

Open questions / follow-ons

• How can this unified semantic-ID space approach be extended to incorporate additional modalities such as audio or video?
• Can the methodology be adapted to dynamically evolving catalogs with rapidly changing modality availability or distribution shifts?
• What is the impact of noisy or low-quality unimodal data on semantic ID consistency and recommendation performance?
• Are there benefits or challenges in jointly learning user representations in this multimodal unified semantic space to complement item semantic IDs?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners focused on recommendation systems or content moderation pipelines that integrate multimodal signals, UnpairGR presents a principled method to incorporate heterogeneous, unpaired item content into discrete, generative representations. This can improve robustness when some modalities are missing or manipulated and reduce reliance on synchronized multimodal data, which is often infeasible at scale. By producing modality-agnostic semantic IDs for items, UnpairGR enables unified downstream reasoning and detection across text and images, which could enhance bot detection models that leverage recommendation behavior.

Furthermore, avoiding modality-specific fallback models simplifies system complexity and reduces potential attack surfaces related to fallback strategies. Practitioners might explore adopting similar shared quantization and modality-reliability weighting techniques to unify signals from diverse and incomplete modalities encountered in security-related user activity profiling or bot-behavior prediction.

Cite

@article{arxiv2608_02477,
  title={ Unpaired Modality-Agnostic Generative Recommendation },
  author={ Weihao Shen and Wei Chen and Fuwei Zhang and Meng Yuan and Yuqin Lan and Guojun Liu and Qingsong Hua and Wei Lin and Fuzhen Zhuang },
  journal={arXiv preprint arXiv:2608.02477},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.02477}
}

Read the full paper

• arXiv:2608.02477 (abstract)
• arXiv:2608.02477 (PDF)