Learning Probabilistic Embeddings for Unsupervised Action Segmentation

Source: arXiv:2607.05263 · Published 2026-07-06 · By Shuai Li, Duc Manh Vu, Juergen Gall

TL;DR

This paper addresses the unsupervised temporal action segmentation problem for long, untrimmed videos, where manual frame-level labels are unavailable. Building on prior methods that combine representation learning with clustering through optimal transport (OT), the authors identify a fundamental limitation: existing methods learn deterministic frame embeddings which can easily get stuck in poor local optima during the iterative training procedure of generating pseudo-labels and refining embeddings. To overcome this, the paper proposes learning probabilistic embeddings modeled as Gaussian distributions per frame to explicitly capture uncertainty, and uses samples from these embeddings during pseudo-label estimation with OT. The approach integrates Graph Convolutional Networks (GCNs) to incorporate temporal context and produce smooth uncertainty-aware embeddings. Evaluations on four standard benchmarks (Breakfast, Youtube Instructional, 50Salads, Desktop Assembly) demonstrate consistent improvements over state-of-the-art unsupervised methods like ASOT and VASOT. Notably, probabilistic embeddings boost mean-over-frames (MoF) accuracy by up to 20.7% and F1-score by up to 19.0% compared to deterministic baselines. The method offers a conceptually simple yet effective framework to overcome overfitting local optima issues in unsupervised temporal action segmentation.

Key findings

• Probabilistic embeddings combined with OT improve MoF by up to +20.7% and F1-score by up to +19.0% over the ASOT baseline across four datasets.
• GCN-based uncertainty estimates outperform MLP and temporal convolution (TCN) for modeling frame embedding distributions, significantly improving segmentation metrics.
• Increasing graph connectivity beyond linking 3 neighboring frames does not improve performance, likely due to GCN oversmoothing effects.
• Using a weighted adjacency matrix (cosine similarity between frame features) in GCN yields better performance than an unweighted graph.
• Monte Carlo sampling with M=3 for the probabilistic embeddings provides a balance between improved accuracy and computational cost; additional samples yield diminishing returns.
• Compared with other stochastic regularization techniques like fixed Gaussian noise or dropout, learned probabilistic embeddings consistently yield better MoF and F1 scores.
• Applying probabilistic embeddings to recent baselines like VASOT also improves segmentation quality, showing the general benefit of modeling embedding uncertainty.
• Qualitative results show probabilistic embeddings better detect reoccurring and short actions and reduce spurious over-segmentation compared to deterministic baselines.

Threat model

n/a — The paper does not focus on adversarial threats or attacker models but rather proposes a representation learning method for unsupervised action segmentation. The main assumption is the lack of any manual frame-level labels or supervision.

Methodology — deep read

The authors formulate unsupervised temporal action segmentation as learning frame embeddings and action prototypes with a joint iterative optimization scheme involving optimal transport (OT). The threat model assumes an unsupervised setting with no access to ground truth labels; the adversary is effectively absent but challenges lie in the complexity of clustering unlabeled frame data. The data comes from four public video datasets—Breakfast, Youtube Instructional (YTI), 50Salads, and Desktop Assembly—using previously extracted frame-level features (IDT or provided features). Data preprocessing includes sampling 256 frames per video and creating temporally weighted graphs connecting adjacent frames.

Instead of deterministic embeddings, the authors propose to learn probabilistic Gaussian embeddings per frame represented by mean (μ) and diagonal covariance (σ^2) vectors. They use a Graph Convolutional Network (GCN) with a weighted adjacency matrix based on cosine similarity of frame features to model temporal context and output both μ and log σ^2 per frame. The reparametrization trick is applied during training: embedding samples z = μ + ϵσ with ϵ ~ N(0,I) are drawn to generate variations.

Optimal Transport (OT), combining Kantorovich and Gromov-Wasserstein formulations, is used to compute pseudo-label soft assignments of frames to actions based on a cost matrix derived from embeddings and temporal priors. The training alternates between estimating OT-based pseudo labels on sampled embeddings and updating parameters by minimizing a Monte Carlo approximated uncertainty-aware cross-entropy loss over M=3 samples. This Expectation-Maximization-like approach incorporates uncertainty to avoid overfitting to noisy pseudo labels.

The GCN consists of a single layer with normalized weighted adjacency connecting 3 frames temporally. The MLP encoder extracts initial frame features. Adam optimizer with learning rate 1e-3 and weight decay 1e-4 is used for training. Action prototypes are initialized by K-means clustering on embeddings, with K the known number of actions per activity.

Evaluation uses standard metrics: mean-over-frames (MoF), segment-level F1-score, and mean Intersection-over-Union (mIoU), computed using Hungarian matching on frame predictions vs ground truth. The method is compared against several state-of-the-art baselines (ASOT, VASOT, CLOT), including ablation studies on embedding type (probabilistic vs deterministic), architecture (GCN vs MLP/TCN), graph construction, sampling count, and comparisons to other stochastic regularization techniques (Gaussian noise, dropout). Qualitative visualizations support the quantitative findings.

The code is publicly released to ensure reproducibility, but the datasets are standard benchmarks with fixed features used from prior work. Although the OT optimization is non-convex, mirror descent is applied efficiently to solve for frame-to-action assignments per training iteration.

Technical innovations

• Introducing probabilistic Gaussian embeddings for frame representations in unsupervised temporal action segmentation to model embedding uncertainty and reduce overfitting to noisy pseudo-labels.
• Using a Graph Convolutional Network with a weighted adjacency matrix to jointly learn smooth frame means and variances encoding temporal context and uncertainties.
• Incorporating Monte Carlo sampling from learned probabilistic embeddings into the optimal transport based pseudo-label estimation within the iterative training loop.
• Employing an uncertainty-aware cross-entropy loss approximated via sampling to stabilize training against pseudo-label noise.
• Demonstrating that sampling multiple embeddings per frame during pseudo-labeling effectively avoids local optima in the optimization compared to deterministic embeddings.

Datasets

• Breakfast — 1712 videos, 48 actions — public benchmark with cooking activity videos, IDT features
• Youtube Instructional (YTI) — 150 videos, 5 activities — public benchmark, pre-extracted features from prior work
• 50Salads (Eval) — 50 videos, 12 merged action classes — public salad preparation dataset, IDT features
• Desktop Assembly (DA) — 76 videos, 22 actions — public benchmark, pre-extracted features used

Baselines vs proposed

• ASOT baseline on Breakfast: MoF = 56.1 vs PEOT = 60.7, F1 = 38.3 vs 40.5
• ASOT baseline on YTI: MoF = 52.9 vs PEOT = 55.4, F1 = 32.1 vs 37.4
• ASOT baseline on 50Salads: MoF = 59.3 vs PEOT = 64.9, F1 = 53.6 vs 58.9
• ASOT baseline on Desktop Assembly: MoF = 70.4 vs PEOT = 71.2, F1 = 68.0 vs 75.8
• VASOT baseline on Breakfast: MoF = 54.5 vs with Prob = 57.2, F1 = 35.3 vs 36.1
• VASOT baseline on 50Salads: MoF = 53.4 vs with Prob = 62.2, F1 = 51.9 vs 53.6
• CLOT reports better mIoU on 50Salads but lower MoF compared to PEOT
• Using GCN probabilistic embeddings improves over MLP or TCN embeddings by up to 2.5% MoF and 2.1% F1 on Breakfast
• Sampling M=3 embeddings outperforms M=1 or M=2 in MoF (+0.7% vs M=1) with negligible runtime increase

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.05263.

Fig 1: Most previous works learn deterministic embeddings as frame embeddings; we

Fig 2 (page 2).

Fig 3 (page 2).

Fig 2: Pipeline of our training scheme. Features of an input video are fed through an

Fig 3: Qualitative results. Comparing ASOT [38], PEOT (ours), and ground truth

Fig 4: Qualitative results. Two examples when using VASOT [2] as baseline. The

Fig 5: Qualitative results of the learned frame-wise representations. Com-

Fig 6: Analysis of the learned uncertainty.

Limitations

• The approach still requires knowledge of the number of actions K per activity for action prototype initialization via K-means, limiting fully unsupervised scenarios.
• The datasets used are all standard public benchmarks with fixed pre-extracted features; effectiveness on raw video or other feature types is untested.
• The model incorporates temporal context via GCN but does not explicitly model complex temporal dependencies like higher-level sequence models or transformers.
• Evaluation uses metrics based on Hungarian matching with ground truth, which may not fully reflect temporal segment boundary quality under real-world noisy or unlabeled conditions.
• Computational complexity increases with the number of embedding samples M during training, though M is kept small (3) here.
• No explicit adversarial or robustness testing against intentional perturbations or domain shifts was reported.

Open questions / follow-ons

• Can probabilistic embeddings be extended to a fully end-to-end trainable pipeline from raw video frames in an unsupervised manner?
• How do learned uncertainty estimates behave under domain shifts or noisy video data, and can they be leveraged for active learning or confidence calibration?
• Can the methodology be adapted or combined with weak supervision or self-supervision signals to further improve segmentation accuracy?
• Are there gains from combining probabilistic embeddings with richer temporal sequence models such as transformers or recurrent networks?

Why it matters for bot defense

While the paper focuses on unsupervised temporal action segmentation in video, the core idea of learning probabilistic embeddings to capture uncertainty and prevent overfitting to noisy pseudo-labels can inspire similar approaches in bot-defense and CAPTCHA systems. For example, uncertainty modeling and probabilistic representation learning could enhance robustness of challenge-response models against adversarial or automated attacks by avoiding premature convergence to spurious features. The use of Graph Convolutional Networks and optimal transport for iterative pseudo-label refinement might also guide new architectures for temporal behavioral analysis in CAPTCHA or bot-detection tasks. However, the domain differences mean the models and data would require careful adaptation.

Bot-defense practitioners should note that probabilistic embeddings allow capturing heteroscedastic uncertainty in representations, a useful property for models that must operate under noisy or adversarially perturbed inputs. Sampling-based training to enforce robustness against label noise can also mitigate overfitting on adversarial or synthetic bot patterns. Thus, the principles here generalize to security-relevant sequential data beyond video, offering a promising research direction for CAPTCHA and bot-defense algorithms that rely on temporal or sequential behavioral signals.

Cite

@article{arxiv2607_05263,
  title={ Learning Probabilistic Embeddings for Unsupervised Action Segmentation },
  author={ Shuai Li and Duc Manh Vu and Juergen Gall },
  journal={arXiv preprint arXiv:2607.05263},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.05263}
}

Read the full paper

• arXiv:2607.05263 (abstract)
• arXiv:2607.05263 (PDF)