SAM2Matting: Generalized Image and Video Matting

Source: arXiv:2606.27339 · Published 2026-06-25 · By Ruiqi Shen, Guangquan Jie, Chang Liu, Henghui Ding

TL;DR

SAM2Matting addresses the longstanding challenge of video matting, which requires both robust high-level target tracking and low-level fine-grained alpha matte estimation. Existing video matting methods rely on expensive, narrowly-scoped, human-centric datasets limiting generalization. This work proposes a novel decoupled paradigm that separately handles temporal consistency via robust Video Object Segmentation (VOS) trackers (SAM2, SAM3) and spatial fine detail matting via newly designed components trained only on diverse image matting datasets. A Region-of-Interest (ROI) Detector identifies matting-critical regions while a Progressive Alpha Predictor progressively refines alpha mattes in a coarse-to-fine manner. The model thus leverages strong pretrained trackers frozen during training to preserve temporal robustness, while matting heads learn pixel-level refinement on rich image data.

This approach achieves state-of-the-art performance on standard image matting benchmarks and, strikingly, surpasses video-supervised baselines in a zero-shot video matting setting, demonstrating unprecedented generalization to both human-centric and complex, in-the-wild videos. It also supports flexible prompt types including masks, points, boxes, and text (for SAM3 variant) with real-time efficiency (up to 40 FPS on 1080p video) and memory usage under 5GB. Ablations validate its architectural and supervision designs as driving the improvements. Fine-tuning on existing video matting datasets improves in-domain results but harms generalization and tracking robustness, motivating the zero-shot methodology. Visual results confirm superior detail preservation, temporal consistency, and robustness to tracking errors or occlusions, showing promise for real-world deployments without costly video alpha annotations.

Key findings

• SAM2Matting (SAM2.1-Tiny) reduces MAD by 11.48 points compared to MAM on P3M-500-NP image matting benchmark.
• In zero-shot video matting on V-HIM60-Hard, SAM2Matting (SAM3) achieves MAD = 3.81 vs MatAnyone2 baseline at MAD = 4.50.
• SAM2Matting attains the lowest dtSSD scores indicating superior temporal consistency inherited from frozen VOS trackers.
• On VideoMatte-SD dataset, SAM2Matting (SAM3) achieves MAD = 4.37 and Grad = 0.27, beating top video-trained baselines.
• ROI Detector outperforms morphological and mask-only ROI strategies with 18.20 MAD vs 29.82 and 20.07 on V-HIM60-Hard.
• Progressive Alpha Predictor's multi-scale refinement and matte-mask consistency penalty reduce artifact scores (MAD from 19.43 to 18.20).
• Fine-tuning SAM2Matting on large video matting dataset V-HIM2K5 improves in-domain MAD from 18.20 to 17.90 but degrades out-of-domain performance.
• SAM2Matting (SAM2.1-Tiny) runs at 40+ FPS on 1080p videos with VRAM under 5GB, outperforming MatAnyone and MatAnyone2 in speed and memory.

Threat model

Not a security paper; adversary model is not defined. The work focuses on robust and general video matting under challenging real-world conditions such as occlusions, fast target motion, and complex backgrounds.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is not explicitly defined as a security threat, but the underlying challenge is robust video matting with stable tracking despite occlusions, rapid motion, complex backgrounds, and semi-transparent or fine-grained regions. The model assumes access only to image-level matting annotations for training the matting heads, not video-level alpha annotations. It leverages pre-trained VOS trackers frozen to preserve temporal robustness, thus assuming the tracker provides relatively accurate initial masks although it can contain errors.

2. Data: Training uses 8 public image matting datasets including I-HIM50K, P3M-10k, CelebAHairMask-HQ, AIM-500, Distinctions-646, AM-2K, UHRIM, and RefMatte. These datasets contain pixel-level alpha mattes for diverse objects, supporting fine-grained alpha learning. No video matting data is used in initial training to enable zero-shot video matting generalization. For evaluation, image matting benchmarks P3M-500-NP, AM-2K test, and PPM-100 are used. Video matting is evaluated zero-shot on V-HIM60 and VideoMatte datasets which have pixel-accurate alpha mattes.

3. Architecture/Algorithm: The framework decouples tracking and matting. It uses a frozen pretrained VOS tracker (SAM2.1-Tiny, SAM2.1-Base+, or SAM3) to generate temporally consistent masks M_t for each video frame t. A newly designed ROI Detector predicts pixel-wise matting-critical regions (semi-transparent, complex boundaries) by integrating multi-scale image features, frame pixels, and tracker masks. This ROI output is thresholded to a binary mask of unknown regions.

From the ROI and mask, a pseudo-trimap is generated assigning pixels to foreground (from mask), background, and unknown (ROI). A Progressive Alpha Predictor takes concatenated inputs of multiscale image features, resized frames, pseudo-trimap, and upsampled alpha matte from the previous scale. It predicts alpha mattes in a coarse-to-fine cascade across 3 scales, iteratively refining the prediction. Each scale has a projection and matting head producing alpha via sigmoid activation.

4. Training Regime: The tracker is frozen during training to preserve temporal consistency. Only the matting modules (ROI Detector and Progressive Alpha Predictor) are trained on images for 5 epochs using AdamW with batch size 32 on 4 NVIDIA A6000 GPUs. Losses include a focal and smooth-L1 loss to train the ROI Detector mask, L1 and Laplacian losses for alpha estimation across all scales (deep supervision), and a matte-mask consistency penalty to prevent hollow regions. Threshold hyperparameters (e.g. ROI confidence θ=0.65, alpha range [0.15, 0.5]) are grid-searched.

5. Evaluation Protocol: Image matting is evaluated using MAD, MSE, Gradient, Connectivity, and SAD metrics on three standard datasets. Video matting evaluation uses the same plus dtSSD to measure temporal stability, all with lower-is-better scoring. Comparison baselines include recent state-of-the-art methods like MatAnyone, MaGGIe, and RVM trained on video matting data. Video matting is evaluated purely zero-shot without video supervision. Ablations test ROI strategies, multi-scale refinement, losses, and fine-tuning effects. Qualitative results demonstrate detail recovery and temporal smoothing. Efficiency is measured by FPS and GPU VRAM usage on VideoMatte benchmark.

6. Reproducibility: Code and models are publicly released at https://github.com/FudanCVL/SAM2Matting. The training datasets are publicly available image matting datasets. The methodology is clearly described allowing reproduction with the same VOS backbones and training setups. No private video matting training data is used in main models.

Example End-to-End: On a video frame, the SAM2 pretrained tracker generates a target mask M_t. The ROI Detector then fuses M_t with multi-scale image features and resized frame pixels to produce a region mask R_t denoting fine matting-critical pixels. A pseudo-trimap T_t classifies pixels to foreground, background, or unknown from M_t and R_t. The progressive alpha predictor inputs the concatenation of image features, T_t, resized frame, and previous scale matte prediction, generating increasingly refined alpha mattes A_t,i at three scales. The final alpha matte A_t is output at full resolution. This separates intricate details like hair strands or translucent objects while maintaining temporal mask coherence across video frames.

Technical innovations

• Decoupling video matting into a two-stage pipeline using frozen VOS trackers for robust temporal consistency and dedicated matting heads trained solely on image datasets for fine spatial alpha estimation.
• Novel ROI Detector that integrates multi-scale image features, the VOS mask, and frame appearance to predict pixel-wise matting-critical regions rather than relying on morphological operations or raw masks.
• Progressive Alpha Predictor using deep supervised coarse-to-fine multi-scale cascaded alpha refinement, guided by pseudo-trimaps derived from ROI and mask predictions.
• Matte-mask consistency loss that prevents hollow foreground regions and anchors alpha matte predictions to tracker masks, improving matting integrity and reducing artifacts.
• Zero-shot video matting without requiring any video matting dataset annotations, enabling robust generalization to human and in-the-wild scenarios.

Datasets

• I-HIM50K — 50000+ images — public image matting dataset
• P3M-10k — 10000 images — public image matting dataset
• CelebAHairMask-HQ — thousands of images — public dataset
• AIM-500 — 500 images — public image matting dataset
• Distinctions-646 — 646 images — public image matting dataset
• AM-2K — 2000 images — public image matting dataset
• UHRIM — 600 images — public image matting dataset
• RefMatte — 3000+ images — public image matting dataset
• V-HIM60 — 60 videos — video matting evaluation dataset
• VideoMatte — 34 videos — video matting evaluation dataset
• V-HIM2K5 — 2500 videos — large-scale video matting dataset for fine-tuning only

Baselines vs proposed

• MAM on P3M-500-NP: MAD = 15.40 vs SAM2Matting (SAM2.1-T) MAD = 3.92
• MatAnyone on AM-2K test: MAD = 10.70 vs SAM2Matting (SAM3) MAD = 4.32
• RVM on VideoMatte-SD: MAD = 6.08 vs SAM2Matting (SAM3) MAD = 4.37 (zero-shot)
• MatAnyone2 on V-HIM60-Medium: MAD = 4.50 vs SAM2Matting (SAM3) MAD = 3.81 (zero-shot)
• ROI Strategy Morphological on V-HIM60-Hard: MAD = 29.82 vs ROI Detector MAD = 18.20
• Fine-tuned SAM2Matting on V-HIM2K5: MAD = 17.90 vs Original zero-shot MAD = 18.20 on V-HIM60-Hard

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.27339.

Fig 1: SAM2Matting enables fine-grained image and video matting across both human and in-the-wild scenarios.

Fig 2: illustrates SAM2Matting, a generalized framework for image and video matting that decouples

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• Zero-shot video matting depends on robustness of the frozen VOS tracker; severe tracking failures can degrade matting quality.
• The model still struggles with extremely complex occlusions or very fast motion beyond evaluated datasets.
• Fine-tuning on video matting datasets harms generalization and tracking robustness, indicating difficulty balancing domain adaptation with generality.
• Evaluations primarily on human-centric and some in-the-wild scenarios; less clear performance on highly cluttered scenes with multiple interacting transparent objects.
• ROIs are binary masks; predicting soft matting-critical region confidences may improve fine boundary estimation but is not explored.
• Model relies on well-curated image matting datasets which have their own domain limitations.

Open questions / follow-ons

• How can tracker and matting components be jointly optimized to improve robustness without sacrificing generalization?
• Can soft ROI prediction or uncertainty modeling further enhance matting of fine semi-transparent regions?
• How to efficiently incorporate large-scale video matting data for adaptation while preventing overfitting to narrow target domains?
• Could self-supervised video data or synthetic datasets enhance zero-shot video matting performance beyond current image-only training?

Why it matters for bot defense

SAM2Matting’s decoupling of high-level temporal tracking from low-level fine-grained matting offers a modular approach relevant for bot-defense systems where reliable foreground extraction from video is necessary. The robust zero-shot generalization and temporal stability without costly video alpha annotations demonstrate a promising direction for scalable, prompt-interactive video analysis frameworks.

The use of an ROI Detector integrating tracking and image features to focus refinement could inspire CAPTCHAs that distinguish genuine user video signals from bots by inspecting semi-transparent or fine boundary regions dynamically. Moreover, the progressive refinement of alpha mattes facilitates accurate extraction of subtle visual cues over time, potentially useful in anti-bot verification tasks requiring high precision video object delineation. The efficiency enabling realtime processing at HD resolutions aligns with deployment constraints in operational bot defense and CAPTCHA systems.

Cite

@article{arxiv2606_27339,
  title={ SAM2Matting: Generalized Image and Video Matting },
  author={ Ruiqi Shen and Guangquan Jie and Chang Liu and Henghui Ding },
  journal={arXiv preprint arXiv:2606.27339},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.27339}
}

Read the full paper

• arXiv:2606.27339 (abstract)
• arXiv:2606.27339 (PDF)