Appearance Pointers -- Multimodal Region Control of Diffusion Transformers

Source: arXiv:2607.19344 · Published 2026-07-21 · By Rahul Sajnani, Yulia Gryaditskaya, Radomír Měch, Srinath Sridhar, Matheus Gadelha

TL;DR

The paper addresses a key limitation in controllable image generation with diffusion transformers (DiTs): enabling precise, region-aware multimodal control over image synthesis. While DiTs can natively take heterogeneous tokens from images and text, they lack an inherent mechanism to specify where and how these tokens influence the generated image. To solve this, the authors introduce Appearance Pointers—compact, modality-agnostic tokens generated by a region correspondence network that link user-supplied region masks with associated text or image prompts. These pointers guide the DiT to apply the correct appearance cues at precise spatial locations, enabling flexible fine-grained control over multiple regions without dramatically expanding the token set or retraining the base DiT model from scratch.

The approach extends DiTs with a spatial aggregation mechanism to fuse multiple regional descriptions into a single denoising pass, supporting simultaneous conditioning with both image and text prompts on the same or different regions. Evaluated on a new synthetic dataset AppearancePointers-37K, which contains paired regional masks, text and image captions, and edited variants, the method achieves state-of-the-art or strong comparable results to modality-specific baselines like MSDiffusion and DreamRenderer across multiple quantitative metrics for regional fidelity, identity preservation, and overall image quality. Qualitative examples show superior spatial adherence and multimodal compositionality. The modular design also supports a variety of editing and generation workflows such as object insertion, pose-conditioning, and multimodal region control.

Key findings

• AppearancePointers achieves top or second-best results on all 6 metrics in text-conditioned region generation on AppearancePointers-37K dataset (Table 2).
• In image-conditioned region generation, AppearancePointers outperforms MSDiffusion and DreamRenderer in region adherence and identity preservation (Table 3).
• AppearancePointers adds only ~3.33% extra parameters (~400M) relative to the base DiT model.
• AppearancePointers enables simultaneous use of image and text prompts on the same region in a single diffusion step, a capability missing in prior approaches.
• The region aggregation transformer reduces token complexity from O(T·(RNreg)^2) to O(R(Nreg/k)^2), enabling efficient multi-region conditioning during inference.
• Including a boundary contour map encoded from aggregated region edges improves fine-grained boundary alignment (Ablation in Table 4).
• AppearancePointers supports diverse tasks including region-controlled generation, object insertion, pose-conditioned generation, and multimodal editing workflows (Table 1).
• Qualitative comparisons demonstrate stronger regional fidelity and semantic correctness on complex scenes involving multi-object and multimodal conditioning (Figures 6 and 7).

Threat model

The paper does not explicitly define a security threat model, as it focuses on improving the controllability of diffusion transformers for legitimate user inputs. The assumed 'adversary' is essentially the intrinsic challenges of multimodal region-conditioned generation, and the work addresses the limitations of model architectures to faithfully localize and compose multimodal prompts. It does not model malicious attackers or adversarial manipulation.

Methodology — deep read

1. Threat Model & Assumptions: The paper assumes a controlled environment where the user provides spatial region masks and corresponding prompts (text, images, or both). The goal is precise regional control during image generation, not adversarial robustness or security threats. The method does not focus on robustness against malicious actors but aims to improve user-driven controllability on legitimate inputs.

2. Data: The authors create a new synthetic dataset AppearancePointers-37K with 37,000+ images featuring multiple objects per scene. Each object is associated with region masks, text descriptions, and reference images from novel views. The dataset is constructed by prompting a large language model (Qwen 3) to produce object- and scene-level captions, then generating images using Flux.1 Dev followed by segmentation with Grounded SAM. Controlled edits (pose, texture, material) are produced and filtered using a vision-language model (InternVL) for consistency. The dataset covers small, medium, and large object scales.

3. Architecture & Algorithm: The core contribution is the Appearance Pointer framework that augments a base diffusion transformer (Flux Kontext DiT) with regional, multimodal conditioning tokens. The system involves two novel modules:

• Region Correspondence Transformer (Φ_RC): For each region i, it takes as input the spatial mask (encoded as an augmented spatial token sequence using coordinate embeddings), local text tokens (T_Pi), and local image tokens (I_Pi). It processes these heterogeneous tokens jointly through lightweight mask transformers and multimodal self-attention blocks, outputting semantic feature maps IM_i (image stream) and TM_i (text stream).
• Region Aggregation Transformer (Φ_A): To avoid quadratic token explosion when conditioning on multiple regions, this transformer aggregates the per-region semantic maps at each spatial patch location across regions. By applying self-attention over the region dimension separately at each patch and prepending a learnable [CLS] token, it condenses multiple region representations into a single appearance pointer per patch for image (I_AP) and text (T_AP) streams.

The appearance pointers thus explicitly link local prompts to spatial masks and condition the base DiT's image and text streams during the denoising process. The DiT inputs include noisy latent tokens, global text prompt tokens, local region image/text tokens, and the appearance pointers.

Additionally, a boundary contour map derived from region edges is encoded and provided to improve spatial precision in generation.

4. Training Regime: The authors train the region correspondence and aggregation modules using a flow-matching diffusion loss, consistent with the Flux Kontext training regime. Training runs on 8 NVIDIA A100 GPUs over 3 days, applying LoRA with rank 128 to new modules. They use Prodigy optimizer with learning rate 1.0. Precise hyperparameter tuning and seed strategies are described in supplementary materials.

5. Evaluation Protocol: Evaluation is performed on held-out splits of AppearancePointers-37K with 500 generated images, each containing on average 5 conditioned regions. Both text-only and image-only regional conditions are tested. Metrics include global CLIP-IQA and CLIP-T scores for overall image quality and prompt alignment; regional metrics DINO-I (region identity), MIoU (region overlap), and region-level CLIP scores measuring adherence to local prompts. Comparisons are made to strong modality-specific baselines (InstanceDiffusion, Seg2Any, DreamRenderer, MSDiffusion). Ablations evaluate the contribution of aggregation and contour guidance modules.

6. Reproducibility: The paper mentions providing pseudocode and supplementary training details, but it is unclear if code or pretrained weights are publicly released. The synthetic dataset is new and may require requests for access. The modular design enables use with off-the-shelf DiT models without retraining from scratch.

Concrete Example End-to-End: Given a scene with multiple masked regions, each associated with a local text prompt and/or reference image, the region correspondence transformer encodes and aligns these inputs into semantic maps specifying how features correspond spatially. These maps are aggregated to form compact appearance pointers, which guide the Flux Kontext DiT during denoising to generate an image respecting global context and precise regional appearance conditions simultaneously. This produces more faithful compositional images than naive guidance with vanilla DiT token concatenation or sequential region editing.

Technical innovations

• Introduction of Appearance Pointers: compact, modality-agnostic tokens that map user-specified spatial masks to local image and text prompts to guide diffusion transformers during generation.
• A Region Correspondence Transformer module that jointly processes spatial masks with heterogeneous local prompts (image and text) to create semantic region embeddings targeting DiT conditioning streams.
• A Region Aggregation Transformer that efficiently consolidates multiple regional semantic embeddings into a single spatially structured appearance pointer canvas, reducing self-attention complexity from quadratic in total tokens to linear in regions.
• First unified framework enabling simultaneous multimodal (image + text) conditioning of multiple regions in a single pass without retraining the base diffusion transformer model.
• Inclusion of a boundary contour encoding mechanism fused at inference time to refine region boundary precision.

Datasets

• AppearancePointers-37K — 37,000+ images with region masks, text and image descriptions, synthetic dataset generated via LLM prompting, Flux.1 Dev rendering, and Grounded SAM segmentation.

Baselines vs proposed

• InstanceDiffusion: Region DINO-I = 35.02 vs AppearancePointers = 56.09 (text-region generation, Table 2)
• DreamRenderer: Region MIoU = 33.84 vs AppearancePointers = 40.35 (text-region generation, Table 2)
• Seg2Any: Region CLIP-I = 89.11 vs AppearancePointers = 90.40 (text-region generation, Table 2)
• MSDiffusion: Global CLIP-I = 89.66 vs AppearancePointers = 92.08 (image-region generation, Table 3)
• DreamRenderer*: Global CLIP-IQA = 74.85 vs AppearancePointers = 95.02 (image-region generation, Table 3)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.19344.

Fig 1: Appearance Pointers enable precise, multimodal, region-aware im-

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• Synthetic dataset may not fully capture the complexity and diversity of real-world scenes; domain transfer not evaluated.
• No adversarial robustness or security threat evaluations are performed; focus is on user-driven controllability.
• Lack of publicly released pretrained models or full codebase limits immediate reproducibility.
• Fine-grained spatial details can be smoothed due to token downsampling in mask transformer and aggregation modules, mitigated but not eliminated by boundary contour guidance.
• Evaluation primarily on quantitative metrics using CLIP and DINO embeddings, which may not fully capture human perceptual quality or semantic alignment.
• No cross-diffusion model architecture comparison; effectiveness shown only on Flux Kontext DiT backbone.

Open questions / follow-ons

• How well do AppearancePointers generalize to real-world datasets beyond synthetic AppearancePointers-37K, including natural images with complex occlusions?
• Can the method be extended or combined with existing attention manipulation or gradient-based guidance for further gains in precision or user control?
• What are the tradeoffs between token compression in aggregation and retention of fine-grained spatial details for more complex or highly detailed regional prompts?
• How can AppearancePointers interface be adapted for video diffusion models or temporally consistent multimodal conditional generation?

Why it matters for bot defense

For bot-defense or CAPTCHA practitioners focused on bot-resistance via multimodal image challenges, this paper's techniques offer a pathway to generate highly controlled synthetic images where specific regions correspond to precise textual or visual cues. Appearance Pointers enable reliable embedding of complex regional instructions into generated scenes, making automated or adversarial recovery of these cues more difficult without exact regional knowledge. The spatial aggregation and multimodal conditioning could inspire CAPTCHA challenges tying human perception to contextually grounded multi-region content. However, the method assumes user-supplied masks and cannot on its own identify suspicious user behavior, so it is one tool in a generative image synthesis arsenal rather than a direct bot detection solution. Deployed responsibly, Appearance Pointers can enhance the fidelity and diversity of challenge images used in bot-mitigation pipelines.

Cite

@article{arxiv2607_19344,
  title={ Appearance Pointers -- Multimodal Region Control of Diffusion Transformers },
  author={ Rahul Sajnani and Yulia Gryaditskaya and Radomír Měch and Srinath Sridhar and Matheus Gadelha },
  journal={arXiv preprint arXiv:2607.19344},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.19344}
}

Read the full paper

• arXiv:2607.19344 (abstract)
• arXiv:2607.19344 (PDF)