ARGUS: Production-Scale Tracing and Performance Diagnosis for over 10,000-GPU Clusters

Source: arXiv:2606.20374 · Published 2026-06-18 · By Jiasheng Zhou, Longbin Zeng, Clavis Chen, Ruiming Lu, Qinwei Yang, Leyi Ye et al.

TL;DR

ARGUS addresses the critical challenge of always-on, fine-grained tracing and real-time performance diagnosis in production-scale GPU clusters exceeding 10,000 GPUs, specifically targeting large language model (LLM) training workloads. Traditional coarse-grained resource monitors fail to localize detailed root causes of fail-slow performance degradations, while existing fine-grained profilers impose prohibitive overheads (5%-30%) and generate massive trace volumes, making continuous deployment infeasible at large scale. ARGUS innovates by decomposing observability along the hierarchical training execution layers—CPU call stacks, framework semantic phases, and GPU kernel traces—enabling comprehensive but low-overhead (<2%) data collection.

ARGUS incorporates a unified data pipeline that compresses raw kernel event traces approximately 3,700x (from 10 MB to 2.7 KB per rank per step) using an online kernel density estimation (KDE)-based clustering approach. A progressive diagnosis framework runs automated detection at multiple granularities—from iteration-level anomaly detection to kernel-level straggler identification—narrowing down tens of thousands of ranks to single-digit suspects for manual root-cause analysis. Deployed on a 10,000+ GPU production cluster for over six months, ARGUS has demonstrated practical fail-slow detection and performance optimization capabilities, successfully diagnosing complex anomalies including compute stragglers, communication link degradations, pipeline bubble amplification, JIT compilation stalls, and hidden compute slowdowns masked by communication symptoms.

Key findings

• ARGUS achieves always-on, fine-grained tracing across CPU call stacks, framework semantics, and GPU kernel execution with less than 2% runtime overhead on 10,000+ GPU clusters.
• Its online statistical compression reduces raw kernel trace size by approximately 3,700 times—compressing 10 MB of raw kernel events per rank per step to around 2.7 KB—enabling scalable cross-rank online anomaly detection.
• The progressive diagnosis framework automatically detects iteration-time anomalies within seconds, attributes straggler ranks using phase-level metrics, and identifies anomalous kernel time distributions within minutes.
• In a 4096-GPU training job, ARGUS detected iteration-time spikes exceeding twice the expected duration, which accounted for about 23,758 GPU-hours wasted (7% of total training compute).
• The clustering approach for kernel duration uses KDE valley detection on log-transformed durations without preset cluster counts and completes within each monitoring window.
• Cross-rank kernel anomaly detection successfully isolates fail-slow ranks by comparing reconstructed CDFs of kernel execution durations using Wasserstein (W1) distance.
• ARGUS was deployed continuously for over six months at Tencent’s production cluster and diagnosed diverse real-world fail-slow cases across multiple software and hardware layers.
• ARGUS improves upon prior systems that either only support coarse metrics at low overhead or fine-grained traces with high overhead and cannot scale to 10,000+ GPUs for real-time cross-rank analysis.

Threat model

Independent fail-slow conditions in components (GPU hardware, communication links, host CPU) degrade synchronous training performance without causing explicit fail-stop errors. The adversary is the complex hardware/software environment causing intermittent performance anomalies during distributed LLM training, not an external attacker. The system assumes trustworthy instrumentation and data collection without adversarial tampering or evasion.

Methodology — deep read

1. Threat Model & Assumptions: ARGUS targets fail-slow adversaries manifesting as intermittent, unpredictable performance degradations in GPU, communication fabric, or host software components synchronously slowing down training ranks without fail-stop errors. The system assumes the training jobs and execution environment are trustworthy, aiming to detect and localize endogenous bottlenecks and anomalies rather than adversarial evasion or tampering.

2. Data: ARGUS continuously collects three complementary observation signals at runtime: (1) CPU call stacks sampled via adapted py-spy to track Python-layer stalls without code instrumentation; (2) framework semantics captured by lightweight CUDA Event instrumentation at entry/exit of training phases (forward, backward, optimizer, communication) to record GPU-side phase durations on correct CUDA streams; (3) GPU kernel execution tracing using NVIDIA CUPTI Activity API to capture per-kernel launch time, duration, and stream. Data is collected from 10,000+ GPUs across a production-scale LLM training cluster with tens of thousands of ranks.

3. Architecture / Algorithm: ARGUS decomposes observability into hierarchy-specific signals independently collected with <2% overhead, enabling always-on tracing. It employs a multi-stage data pipeline where local Trace Producers send raw events to Processor per host. Processor filters, normalizes, and converts data into Perfetto trace format for object storage and performs real-time online statistical compression of kernel traces into compact kernel density estimation (KDE)-based clusters per (kernel, stream, rank) tuple. KDE valley detection on log-transformed durations automatically finds multimodal clusters without preset K, extracts count, median (p50), and 99th percentile (p99) statistics per cluster, compressing 104–105 kernel events per GPU per minute to several kilobytes per rank per minute. A tiered storage system supports low-latency query, alerting (metrics to Prometheus/Grafana), and offline deep analysis.

The progressive diagnosis framework implements parallel anomaly detection levels: L1 (iteration-level anomaly detection using ratio-gated jitter and change-point detection), L2 (cross-rank straggler detection based on semantic phase durations with parallelism-group-aware routing), L3 (kernel-level fail-slow ranking by comparing reconstructed CDFs from KDE summaries across ranks via Wasserstein distance), with L4/L5 providing manual deep-dive trace and CPU stack analysis.

4. Training Regime: Not applicable as ARGUS is a monitoring system embedded in production training runs. The system was deployed on a 10,000+ GPU cluster continuously for over six months.

5. Evaluation Protocol: ARGUS evaluates overhead (<2%) and compression ratio (~3,700x) directly measured on production clusters, as well as detection timeliness at seconds to minutes granularity. Multiple real-world case studies demonstrate successful detection and root cause analysis of complex fail-slow issues. Performance gains are shown by avoided GPU-hours lost to fail-slow and improved training throughput after diagnosis.

6. Reproducibility: The system integrates open source tools (py-spy, CUPTI, Perfetto) with proprietary components and tiered storage, but detailed code release status is not stated. Kernel clustering uses publicly cited KDE-based methods. Data sources are proprietary Tencent LLM cluster traces, not publicly released. The overall architecture and algorithms are fully detailed to enable reproduction.

Example end-to-end flow: During production training, ARGUS’s Trace Producer samples CPU stack snapshots and inserts CUDA Events around training phases while continuously tracing all GPU kernel launches. The Processor receives these raw traces and performs KDE-based clustering on kernel durations in each time window, computing cluster summaries. Metrics and compressed data stream to Metric Storage for live Grafana dashboards and anomaly triggers. L1 detects an iteration time spike; L2 identifies the straggler rank via phase duration outlier scores in that parallelism group; L3 drills down to isolate the degraded kernel by measuring W1 distances of kernel duration CDFs across ranks. The user can then load corresponding Perfetto traces and CPU stacks from Object Storage for manual root-cause analysis and remediation.

Technical innovations

• Hierarchical decomposition of always-on tracing into CPU stack sampling, lightweight framework semantic instrumentation, and GPU kernel execution tracing lowers overhead below 2% for 10,000+ GPU clusters.
• Online statistical compression of voluminous kernel traces using KDE-based valley detection clustering on log-duration samples achieves ~3,700x data reduction while preserving multimodal runtime behavior.
• Progressive multi-granularity diagnosis framework automatically narrows fail-slow search scope from iteration-level anomalies to specific degraded kernels using cross-rank, kernel-level statistical comparisons with Wasserstein distance.
• Parallelism-group-aware routing ensures anomaly attribution respects distributed training parallel roles, improving straggler detection precision at scale.

Datasets

• Tencent production LLM training cluster traces — 10,000+ GPUs — proprietary, not publicly released

Baselines vs proposed

• Greyhound: overhead < 2%, granularity coarse, no kernel-level analysis; ARGUS: overhead < 2%, kernel-level granularity with online cross-rank anomaly detection
• MegaScale: triggered kernel profiling with 5-30% overhead, phase-level granularity; ARGUS: always-on, < 2% overhead, continuous kernel-level anomaly detection
• EROICA: triggered kernel profiling, partial coverage; ARGUS: continuous profiling with scalable compression enabling real-time detection
• FLARE: restricted operator coverage and limited granularity; ARGUS: general kernel tracing enabling diagnosis of diverse anomaly types

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.20374.

Fig 3: Overall architecture of ARGUS.

Fig 7: Kernel statistics anomaly detection workflow.

Fig 10: Case 1: Grafana heatmap of per-rank maximum operator

Fig 11: Case 2: 𝑊1 distance matrices for three communication

Fig 12: Case 2: L4 Perfetto trace of communication kernels.

Fig 13: Case 3 (a): L4 Perfetto trace of one PP group

Fig 14: Case 3 (b): L4 Perfetto trace of ranks at the

Fig 16: Case 5: Heatmap of per-rank max duration. The x-axis

Limitations

• The compression method is lossy and while preserving key statistics, may obscure rare micro-anomalies occurring outside dominant KDE clusters.
• Code and dataset are proprietary, limiting independent reproducibility and external validation.
• Evaluation focuses on production deployment and case studies rather than controlled experimental benchmarks or adversarial fault injection.
• No explicit discussion of handling adversarial evasion or malicious noise in tracing data.
• Assumes static training parallelism groupings known for routing, which may be complex in elastic or dynamic jobs.
• Manual deep-dive analysis is still required after automated narrowing; fully automated root cause classification not addressed.

Open questions / follow-ons

• How to fully automate root cause classification and remediation decision-making beyond progressive narrowing?
• Can the statistical compression preserve more detailed rare pathological kernel events without increasing overhead?
• How generalizable is ARGUS’s architecture for other HPC workloads beyond LLM training?
• What is the tradeoff between compression fidelity and detection sensitivity under various anomaly types?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, ARGUS illustrates a compelling approach to finely instrument and analyze massive distributed workloads with tight overhead constraints, balancing observability depth against scalability. ARGUS’s decomposition of signals and layered progressive diagnosis could inspire similar hierarchical tracing and anomaly detection strategies for large-scale automated attack detection systems, where root causes of performance disruptions or suspicious behaviors may be intermittent and masked. The KDE-based online compression method is also noteworthy for enabling real-time cross-instance comparison under massive data volume constraints, a challenge common in botnet detection or CAPTCHA solving behavior analysis.

Unlike classical bot defense relying on coarse metrics or intermittent sampling, ARGUS’s architectural principles highlight the value of lightweight continuous instrumentation coupled with scalable statistical summarization and progressive analysis to pinpoint subtle, multi-layer anomalies. However, ARGUS’s domain is very different technically from CAPTCHA solving, so direct application requires careful adaptation. The methodology for diagnosing fail-slow conditions in distributed systems may inspire robustness and interpretability approaches in bot-defense monitoring systems.

Cite

@article{arxiv2606_20374,
  title={ ARGUS: Production-Scale Tracing and Performance Diagnosis for over 10,000-GPU Clusters },
  author={ Jiasheng Zhou and Longbin Zeng and Clavis Chen and Ruiming Lu and Qinwei Yang and Leyi Ye and Ray Ying and Key Zhang },
  journal={arXiv preprint arXiv:2606.20374},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.20374}
}

Read the full paper

• arXiv:2606.20374 (abstract)
• arXiv:2606.20374 (PDF)