Repurposing CLIP to Localize at Pixel Level

Source: arXiv:2607.05253 · Published 2026-07-06 · By Jiaxiang Fang, Shiqiang Ma, Jing Wang, Siyu Chen, Fei Guo, Shengfeng He

TL;DR

This paper addresses the challenge of adapting the large-scale vision-language model CLIP—which excels at image-level open-set localization—to precise pixel-level dense prediction for segmentation tasks. Prior approaches struggle due to CLIP's global feature bias and coarse localization outputs that lead to noisy or incomplete segmentations, especially for unseen categories. The authors propose CLIPix, a novel framework that traces back CLIP's classification logits to extract class-specific attentive activation maps as initial pixel-level localization cues. CLIPix then employs a Noise-Resistant Correction strategy that iteratively refines these cues by reweighting patch embeddings and updating class tokens, reducing noise and enhancing target specificity. Additionally, a Localization Embedding strategy injects these refined localization maps into the image features through concatenation and cross-attention, enabling detailed, high-resolution segmentation. The approach retains CLIP's strong generalization to unseen classes and supports zero-shot open-set segmentation. Extensive experiments on PASCAL-5i and COCO-20i datasets demonstrate that CLIPix substantially outperforms state-of-the-art zero-shot and few-shot methods (by up to ~26.5% mIoU) and is competitive even against foundation models relying on prompt tuning or support samples, while being more efficient and suitable for edge deployment.

Key findings

• CLIPix improves zero-shot mIoU on PASCAL-5i by 21.3% and on COCO-20i by 26.5% compared to prior state-of-the-art zero-shot methods (Tables I and II).
• The Noise-Resistant Correction strategy enables iterative refinement of pixel-level localization by reweighting image embeddings and class tokens, reducing background noise and false positives.
• Localization Embedding fuses refined localization maps with image features via concatenation and cross-attention, enhancing fine details such as whole-object coverage beyond partial keypoint responses.
• CLIPix works well with multiple backbones including CLIP-ViT-B/16, MobileNetV2, and EfficientNet-B4, demonstrating strong performance with lighter weight encoders for resource-constrained scenarios.
• CLIPix achieves FBIoU (foreground-background intersection over union) improvements of 19.4% and 18.9% on PASCAL-5i and COCO-20i, respectively, showing precise foreground-background separation.
• Unlike methods relying on external large models (GPT-4, SAM, LLaVA), CLIPix requires no auxiliary support samples or rich textual prompts and thus offers lower computational cost.
• CLIPix uses simple text prompts ('a photo of [class]') to generate text embeddings, avoiding risks of overfitting seen in strong class-aware contrastive training methods like LSeg and SAZS.
• Qualitative results show CLIPix identifies complete object regions even for unseen classes, outperforming affinity-matrix baseline approaches prone to false positives.

Threat model

An adversary is conceptualized as the open-set nature of image inputs containing unseen object classes not present during training. The model must generalize zero-shot without labeled masks or support examples for these classes. The adversary cannot manipulate model parameters or inject deceptive prompts but contaminations arise naturally from global feature biases leading to noisy localization. The aim is to robustly segment arbitrary objects despite this unknown class space.

Methodology — deep read

The authors propose a framework to repurpose CLIP's natural image-level classification capability for dense pixel-level localization and segmentation under open-set conditions, i.e., for arbitrary unseen classes.

1. Threat model & assumptions: The adversary is conceptualized as an open-set environment where at test time input images may contain object classes not seen during training. The model cannot rely on closed-set labels and must generalize zero-shot. The approach assumes no access to ground-truth masks for unseen classes.

2. Data: Training and evaluation use standard binary open-set semantic segmentation datasets PASCAL-5i and COCO-20i. The datasets are split into folds where training classes and test classes are disjoint to assess open-set generalization. Ground-truth segmentation masks accompany images for evaluation; typical metrics like mean Intersection over Union (mIoU) and foreground-background IoU (FBIoU) measure performance.

3. Architecture and algorithm:

• The backbone is a pre-trained CLIP ViT-B/16 visual encoder with frozen weights.
• Given an input image I and a text prompt template 'a photo of [cls]', the CLIP text encoder produces a text feature vector V_text.
• The image is encoded into patch embeddings E and a class token C. CLIP produces image-level features F_visual which are pooled to V_visual.
• The classification score S is the cosine similarity between V_text and V_visual.
• To localize at pixel level, the method traces back gradients of the classification score w.r.t. intermediate visual features to obtain class-specific activation maps L1, representing initial rough localization.
• Because L1 is noisy due to CLIP's global bias towards whole-image features, a Noise-Resistant Correction strategy refines it iteratively: a) L1 is reshaped and used to reweight patch embeddings E, updating them to E_new via Hadamard product. b) A pseudo-target prototype is constructed by combining the original class token C and weighted embeddings, updating the class token to C_new. c) The refined inputs X_new = [E_new, C_new] are fed again through the visual encoder to generate cleaner localization maps L2.
• The Localization Embedding strategy then integrates L1, L2, and image features: a) Localization prototypes P1 and P2 are pooled features weighted by L1 and L2, respectively. b) These prototypes are expanded and concatenated with initial image features F_vis, then passed through convolution and ReLU to create intermediate features F_vis_middle capturing object detail relations. c) Cross-attention is performed between F_vis_final (concatenation of F_vis, F_vis_middle, L1, L2) and the weighted F_vis_middle⊙L2 to propagate local to global information, producing an attention map M. d) The attention map M and prototypes are combined to produce refined output features F_out.
• A lightweight transformer decoder and classifier decode F_out into pixel-level segmentation masks.
• The entire approach freezes the main CLIP parameters, only training the decoding components with cross-entropy loss.

4. Training regime:

• Optimizer: AdamW with momentum 0.9, learning rate 0.0001.
• Batch size 16.
• Training epochs: 200 for PASCAL-5i, 50 for COCO-20i.
• Hardware: single NVIDIA Tesla V100 GPU.
• Backbones experimented with include ViT-B/16, ResNet50, MobileNetV2, EfficientNet-B4.

5. Evaluation:

• Metrics: mean Intersection over Union (mIoU) and FBIoU.
• Baselines include prior zero-shot and few-shot semantic segmentation methods like SAZS, LSeg, PMGA-Net, and foundation models such as SAM variants.
• Quantitative evaluation uses held-out test folds disjoint from training classes to assess zero-shot generalization.
• Ablation includes comparing affinity matrix localization vs CLIPix localization to demonstrate noise reduction.
• Qualitative visualization of segmentation masks on PASCAL-5i and COCO-20i shows detailed object coverage.

6. Reproducibility:

• Code is publicly available at github.com/aqingaqinghh/CLIPix.
• CLIP backbone weights are publicly pre-trained.
• Dataset splits follow standard segmentation benchmarks.
• Implementation details of optimization and hyperparameters are provided.

A concrete example: For an input image and class 'dog', the CLIP text encoder maps 'a photo of dog' to V_text. The image encoder generates patch embeddings and a class token. By backpropagating the classification score, CLIPix extracts an activation map L1 highlighting dog regions. L1 serves as a spatial weight to modulate patch embeddings and update the class token, producing refined embedding input X_new. Feeding X_new through CLIP again yields a cleaner localization map L2. Localization embeddings constructed from L1, L2, and image features enrich the feature maps, which are then processed by a transformer decoder and a classifier to produce the final pixel-wise dog mask prediction, which is then evaluated against ground truth.

Technical innovations

• Tracing back CLIP's visual-language classification logits to extract class-specific attentive activation maps as initial pixel-level localization cues.
• A Noise-Resistant Correction strategy that iteratively refines localization by reweighting patch embeddings and updating class tokens to reduce global bias noise.
• A Localization Embedding strategy that combines refined localization maps and image features through concatenation and cross-attention to enhance detailed object segmentation.
• Integration of these methods enables zero-shot generalizable pixel-level segmentation retaining CLIP’s classification generalization without extra support samples.

Datasets

• PASCAL-5i — 20 classes with 4 folds — Expanded PASCAL VOC 2012 with SBD annotations, public benchmark
• COCO-20i — 80 classes with 4 folds — Derived from MS COCO, public benchmark

Baselines vs proposed

• SAZS zero-shot mIoU (PASCAL-5i) = 55.5% vs CLIPix = 80.7%
• LSeg zero-shot mIoU (PASCAL-5i) = 47.4% vs CLIPix = 80.7%
• PMGA-Net zero-shot mIoU (PASCAL-5i) ~70.6% vs CLIPix = 80.7%
• CLIPSeg zero-shot mIoU (COCO-20i) ~34.9% vs CLIPix = 61.8%
• DSV-LFS few-shot mIoU (COCO-20i) ~71.9% vs CLIPix zero-shot mIoU = 61.8%, noting DSV-LFS uses multi-model ensemble
• CLIPix FBIoU (PASCAL-5i) = 88.4% vs PMGA-Net FBIoU = 80.0%
• CLIPix outperforms SegGPT on COCO-20i by 5.7% mIoU despite SegGPT training on all classes

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.05253.

Fig 1: The overview of our CLIPix framework. CLIPix repurposes CLIP for precise pixel-level localization while preserving its inherent generalization

Fig 2: illustrates an overview of our method. First, we

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 4).

Fig 7 (page 4).

Fig 8 (page 4).

Limitations

• Experimental evaluation focuses on binary open-set semantic segmentation; applicability to multi-class or panoptic segmentation is unclear.
• CLIPix relies on frozen CLIP backbones; performance is tied to backbone quality and may be less competitive with ever-larger multimodal models.
• The Noise-Resistant Correction assumes the quality of initial activation maps; severely occluded or small objects may still pose localization challenges.
• The method does not explicitly address adversarial robustness or evaluate against adversarially perturbed inputs.
• Localization embedding and correction are implemented with handcrafted hyperparameters (e.g., α1=α2=0.5) whose sensitivity and tuning details are not fully explored.
• Evaluation is on publicly available datasets; real-world distribution shifts or domain adaptation scenarios are not investigated.

Open questions / follow-ons

• How well would CLIPix scale and adapt to multi-class or panoptic segmentation involving multiple overlapping objects?
• What is the robustness of the localization correction against occlusion, clutter, or adversarial perturbations?
• Can the Noise-Resistant Correction and Localization Embedding be optimized jointly end-to-end rather than as post-hoc refinements?
• How sensitive is the method to the choice of text prompts and can prompt engineering improve localization further?

Why it matters for bot defense

For bot-defense and CAPTCHA engineers, CLIPix presents an insightful example of repurposing large vision-language models for pixel-level object localization without requiring closed vocabularies or support images. Its method of backpropagating classification logits to generate fine-grained spatial cues and refining them via noise correction mechanisms highlights practical approaches to overcoming global feature bias—a common issue in multimodal AI. Such dense, zero-shot segmentation capabilities can improve bot-detection systems relying on scene understanding or challenge generation that depends on object recognition at fine scales, allowing more robust and generalizable systems against bots trained to exploit coarse features. Furthermore, CLIPix’s efficiency relative to large ensembles or prompt-heavy approaches points to practical deployments in resource-constrained environments typical of CAPTCHA image challenges.

Cite

@article{arxiv2607_05253,
  title={ Repurposing CLIP to Localize at Pixel Level },
  author={ Jiaxiang Fang and Shiqiang Ma and Jing Wang and Siyu Chen and Fei Guo and Shengfeng He },
  journal={arXiv preprint arXiv:2607.05253},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.05253}
}

Read the full paper

• arXiv:2607.05253 (abstract)
• arXiv:2607.05253 (PDF)