ClouDens: Operational Context-Aware Anomaly Detection for Large-scale Cloud System Monitoring
Source: arXiv:2607.18127 · Published 2026-07-20 · By Thu T. H. Doan, Mohammad Saiful Islam, Andriy Miranskyy, Ngoc-Thanh Nguyen, Rogardt Heldal, Patrizio Pelliccione
TL;DR
This paper addresses the challenge of anomaly detection in large-scale cloud systems (LCSs), where continuous telemetry logs produce extremely high-dimensional, heterogeneous, and sparse multivariate time series data. Detecting anomalies early and accurately in this context is difficult due to complex operational dependencies among cloud microservices and the sparsity caused by intermittently active services. To tackle these challenges, the authors first conduct an empirical study on telemetry data from the IBM Cloud Console platform, then propose ClouDens, an anomaly detection framework that explicitly leverages operational-context attributes embedded in telemetry logs to inform modeling. ClouDens partitions telemetry features into semantically meaningful subsets, constructs context-aware graphs encoding operational service dependencies, and applies spatio-temporal graph neural networks (ST-GNNs) for forecasting-based anomaly detection.
Key findings
- ClouDens achieves higher NAB scores on count-based telemetry features compared to a baseline GRU model, indicating more accurate and earlier detection with broader anomaly coverage.
- Telemetry decomposition by HTTP status codes and aggregation types improves detection by focusing on homogeneous feature groups.
- Operational-context graph modeling effectively captures service dependencies and improves forecasting accuracy for anomaly detection.
- Sparsity imputation combined with a missing-value node feature improves handling of telemetric sparsity common in LCSs.
- Ensemble post-processing of anomaly scores from multiple telemetry subsets increases anomaly coverage without significantly increasing false alarms.
- Detection performance is sensitive to scoring strategies (e.g., Mahalanobis distance vs likelihood) and sliding window sizes for forecasting.
- Localized anomalies predominantly affect only subsets of telemetry features rather than the entire system simultaneously.
- Context-aware spatio-temporal modeling better captures the joint temporal and spatial dynamics of API activities in LCSs than standalone sequence models.
Threat model
The adversary here corresponds to system faults or performance degradations that manifest as anomalous telemetry patterns in cloud services. The assumption is that anomalies indicate failures or unexpected behavior rather than intentional adversarial manipulation. The detection model does not consider an adaptive attacker that can deliberately manipulate telemetry to evade detection.
Methodology — deep read
The authors start from a threat model where anomalies correspond to unexpected operational behaviors in cloud telemetry logs reflecting failures or degradations, while assuming no adversarial manipulations. The primary data is the IBM Cloud Telemetry Dataset, collected from the IBM Cloud Console production environment, containing millions of telemetry records representing RESTful API calls, with operational context metadata (deployment location, communication role, HTTP method, status code, endpoint ID) and numerical performance metrics (request counts, response times aggregated by count, avg, min, max). The dataset includes labeled anomaly intervals derived from incident reports and operator logs. Data preprocessing includes decomposing the raw telemetry matrix into semantically coherent subsets by grouping features based on HTTP status codes (e.g., 4xx, 5xx) and aggregation types, to reduce dimensionality and improve interpretability. Missing telemetry values caused by sparse API activities are imputed, and an additional binary missing-value node feature is added to indicate imputed values.
To model spatio-temporal dependencies, ClouDens constructs weighted undirected context-aware graphs per telemetry subset where nodes correspond to telemetry features and edges reflect shared operational-context attributes, with edge weights proportional to the degree of similarity (e.g., sharing deployment location, service component, endpoint). The forecasting backbone is a Spatio-Temporal Graph Neural Network (ST-GNN) that jointly learns temporal patterns and spatial service dependencies. Node features are the telemetry metric values plus missing-value flags, and the model predicts future telemetry feature vectors in a sliding window fashion. Loss is computed as mean squared error between predicted and true metric values.
Anomaly scoring transforms forecasting errors into anomaly likelihoods or Mahalanobis distances, followed by thresholding to identify anomalous timestamps. An ensemble step merges detections from different telemetry subsets using a one-vote (logical OR) strategy to improve anomaly coverage while controlling false positive rates. Various post-processing options (scoring functions, ensembling approaches, sliding window sizes) are empirically compared.
The training regime details such as batch sizes, epochs, seed strategy, or hardware configurations are not fully specified in the truncated text. Evaluation uses the NAB (Numenta Anomaly Benchmark) score and detection coverage metrics to compare ClouDens against a GRU-based baseline and assess ablations on scoring and imputation. Experiments focus on the IBM dataset’s count-based telemetry features, highlighting improved early detection and anomaly coverage due to context-aware graph modeling and decomposition. The authors provide a replication package with implementation code, experiment scripts, and dataset references for reproducibility.
Technical innovations
- Introducing domain-guided telemetry feature decomposition based on operational metadata to partition high-dimensional telemetry into semantically coherent subsets.
- Constructing context-aware weighted graphs from operational-context attributes to explicitly model service dependencies in the anomaly detection model.
- Applying Spatio-Temporal Graph Neural Networks for forecasting-based anomaly detection tailored to cloud telemetry's complex spatial-temporal dependencies.
- Incorporating a binary telemetry sparsity indicator as an auxiliary node feature to handle sparsity and improve forecasting robustness in sparse telemetry.
- Using ensemble post-processing of anomaly scores from decomposed telemetry subsets with a one-vote fusion strategy to increase anomaly coverage.
Datasets
- IBM Cloud Telemetry Dataset — millions of telemetry records with multivariate time series and contextual metadata — publicly available on Zenodo
Baselines vs proposed
- GRU-based anomaly detection: NAB score = lower than ClouDens
- ClouDens (proposed): higher NAB score on count-based telemetry features with earlier detection and broader anomaly coverage
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.18127.
Fig 1: Overview of telemetry-driven anomaly detection for
Fig 2 (page 1).
Fig 4: Sum of count of 5xx count subset as MTS and several anomalies detected by ClouDens.
Fig 5: An illustration of detected anomalies from GRU and ClouDens.
Fig 6: Detected anomalies captured by each subset.
Fig 7: Impact of sliding windows and telemetry sparsity
Fig 8: Impact of sliding windows on Training/Inference Time.
Limitations
- No explicit evaluation against adversarial fault injection or deliberate evasion; assumptions rely on anomalies reflecting natural faults.
- Training regime details, such as hyperparameters and hardware, lack full disclosure.
- Generalizability beyond the IBM Cloud Telemetry Dataset is not tested; effectiveness on other cloud platforms remains unverified.
- Impact of distribution shifts or evolving telemetry distributions over time is not evaluated.
- Sparsity imputation methods and their sensitivity are only partially studied.
- The framework does not introduce novel ST-GNN architectures but adapts existing models to the telemetry context.
Open questions / follow-ons
- How would ClouDens perform under evolving telemetry distributions or concept drift inherent in dynamic large-scale cloud systems?
- What are the effects of different sparsity imputation techniques and their impact on anomaly detection accuracy?
- Can adversarial robustness be integrated into the framework to handle stealthy, crafted anomalies?
- What is the impact of different ST-GNN architectures or attention mechanisms tailored for cloud telemetry?
Why it matters for bot defense
This work is highly relevant to bot-defense and CAPTCHA practitioners interested in anomaly detection in telemetry data from large, distributed cloud systems. The use of operational context to inform model construction and the decomposition of high-dimensional telemetry into coherent subsets provide practical guidance for designing effective monitoring systems. The emphasis on spatio-temporal graph neural networks enables capturing dependencies that simpler sequence models overlook, resulting in earlier and more accurate detection of localized anomalies. However, the approach targets fault detection in cloud infrastructure telemetry and does not address adversarial traffic or bot behavior directly. Still, the method's insights on contextual modeling, sparsity handling, and ensemble strategies could inspire more robust detection pipelines for abnormal traffic or automated interactions in cloud environments that underpin CAPTCHA platforms. Practitioners should consider the challenges of telemetry sparsity, high dimensionality, and spatial dependencies when engineering security and bot-detection observability.
Cite
@article{arxiv2607_18127,
title={ ClouDens: Operational Context-Aware Anomaly Detection for Large-scale Cloud System Monitoring },
author={ Thu T. H. Doan and Mohammad Saiful Islam and Andriy Miranskyy and Ngoc-Thanh Nguyen and Rogardt Heldal and Patrizio Pelliccione },
journal={arXiv preprint arXiv:2607.18127},
year={ 2026 },
url={https://arxiv.org/abs/2607.18127}
}