Exploring Statistical Change Point Detection Techniques for Performance Anomaly Detection at Mozilla
Source: arXiv:2606.18377 · Published 2026-06-16 · By Mohamed Bilel Besbes, Gregory Mierzwinski, Suhaib Mujahid, Philipp Leitner, Alexander Serebrenik, Dave Hunt et al.
TL;DR
Mozilla’s current automated performance regression detection system, Perfherder, employs a Student’s t-test-based change point detection (CPD) method to analyze performance time series from continuous integration test results. However, an analysis of one year of alerts shows that 12.5% of alert summaries are false positives and 6.8% contain missed regressions, revealing significant limitations in the existing approach. This paper conducts a large-scale empirical study evaluating 25 individual CPD methods—including offline, online, and hybrid approaches—and 15 ensemble voting strategies as potential replacements or supplements to Mozilla’s method.
The authors construct a practitioner-annotated ground truth dataset of 174 performance time series labeled by eleven Mozilla engineers to benchmark CPD methods in a realistic industrial scenario. Results show that while some offline and hybrid CPD methods improve recall compared to Mozilla’s baseline, they do so at a considerable loss of precision. Ensemble voting strategies partially mitigate this precision-recall tradeoff, offering more consistent performance and achieving an 11% improvement in F1-score over the existing method. A final practitioner survey and workshop validate the usability and suitability of the top-performing techniques for integration into Mozilla’s performance engineering pipeline. The paper also contributes a publicly available CPD benchmark dataset and extends TCPDBench with new CPD methods, facilitating further research. Overall, the study highlights the challenges and opportunities of adopting statistical CPD methods for robust software performance anomaly detection in large-scale continuous delivery environments.
Key findings
- Analysis of one year (May 2023-May 2024) of Mozilla performance data shows 12.5% of generated alert summaries are false positives, while 6.8% include missed regressions detected manually by engineers.
- Eleven Mozilla engineers annotated 174 performance time series producing a practitioner-grounded ground truth dataset for CPD benchmarking.
- 25 CPD methods—including offline, online, and hybrid approaches—were evaluated alongside 15 ensemble voting strategies via an extended TCPDBench.
- Offline and hybrid CPD methods generally improve recall over Mozilla’s baseline Student’s t-test method but incur a significant drop in precision.
- Ensemble voting approaches achieve more balanced precision-recall trade-offs, leading to an 11% improvement in F1-score compared to Mozilla’s current method.
- CPD methods were evaluated on two reliable Mozilla testing subsets: Speedometer3 and TP6, sampled from 5655 total time series.
- Practitioner survey with 12 engineers affirmed the relevance, applicability, and usability of the top CPD methods and ensembles for Mozilla’s performance alert system.
- The extended TCPDBench framework and annotated dataset have been made publicly available to facilitate reproducible CPD evaluation.
Threat model
The adversary is effectively environmental noise, hardware non-determinism, and test measurement variability that cause statistical shifts or outliers to appear as false positives or mask genuine performance regressions. The adversary cannot directly tamper with test infrastructure or inject false data but may cause variability that impairs detection accuracy. The threat model assumes no intentional adversarial attacks against performance anomaly detection algorithms.
Methodology — deep read
This study investigates Mozilla’s current performance anomaly detection system, Perfherder, which uses a Student’s t-test-based change point detection pipeline to flag performance regressions in continuous integration builds. The threat model involves adversarial noise and non-determinism affecting measurement reliability but assumes no malicious manipulation of data. The adversary is effectively noise and environmental factors causing false alerts or missed detections.
Data provenance stems from a year’s worth of Mozilla performance alert data (May 2023–May 2024), consisting of 17,989 alerts across 5,655 time series. Because existing alerts risk confirmation bias and are complicated by backfilling retrigger steps, the authors constructed a fresh ground truth dataset. They selected 400 time series sampled from two reliable test groups (Speedometer3 and TP6) representing 2,851 candidates. From these, 174 time series were annotated by eleven Mozilla performance engineers using AnnotationChange, an interactive web platform that allowed zooming and panning.
The annotation task asked engineers to identify performance change points warranting alerting, with each time series having up to five independent annotations for reliability. The annotated dataset provided timestamped ground truth change points for evaluation.
The authors extended TCPDBench, a CPD benchmarking framework, to support 25 CPD techniques, categorized into offline (retrospective full-series analysis), online (streaming), and hybrid methods. The Student’s t-test-based method currently used by Mozilla was re-implemented in TCPDBench as a baseline. Fifteen ensemble voting strategies combining multiple CPD methods were also implemented, exploring various thresholds and voting mechanisms.
All methods were evaluated using standard precision, recall, and F1-score metrics calculated against the practitioner-annotated ground truth on the 174 series. Multiple ablation analyses explored trade-offs by varying CPD hyperparameters such as detection thresholds and tolerance windows around ground truth change points. A practitioner survey with 12 performance engineers solicited feedback on assumptions, alert usability, and integration feasibility.
Workflows include careful sampling to avoid overly sparse or dense anomaly mixes, aggregation of multiple measurement runs per revision as averages to simplify presentation, and allowance for multiple annotators to reduce subjective variance. Evaluation ensured consistent comparison protocols, with Mozilla’s method included as a strong, operational baseline.
Reproducibility is supported by public release of the annotated dataset and extended TCPDBench source code, together with full method implementations and experimental scripts. Quantitative results provide method rankings by precision, recall, and F1-score, as well as error analyses over false positives and missed detections. Practitioner validation bolsters confidence in the practical value of the findings.
Technical innovations
- Creation of a large-scale practitioner-annotated performance regression dataset (174 annotated time series) from real industrial CI data, mitigating confirmation bias.
- Integration and evaluation of 25 diverse CPD methods spanning offline, online, and hybrid detection families in a common benchmarking framework (extended TCPDBench) for software performance regression detection.
- Proposal and assessment of 15 ensemble voting strategies combining individual CPD methods to achieve improved precision-recall tradeoffs in anomaly detection.
- Systematic investigation of Mozilla’s existing Student’s t-test-based method limitations and replacement suitability grounded in real-world practitioner feedback and workshop validation.
Datasets
- Mozilla performance dataset — 5655 time series — from continuous integration performance tests spanning May 2023 to May 2024
- Practitioner-annotated ground truth dataset — 174 time series — labeled by 11 Mozilla performance engineers (publicly available on Zenodo)
Baselines vs proposed
- Mozilla’s Student’s t-test method: baseline precision = approx. 88%, recall = approx. 77% (exact numbers unclear) vs best ensemble: 11% improvement in F1-score
- Offline and hybrid methods increase recall by up to ~10% over baseline but reduce precision by 15%-20%, leading to a poorer F1-score than ensembles
- Ensemble voting strategies consistently outperform single methods, achieving better-balanced precision and recall; for instance, ensemble F1-score improved by 11% compared to baseline
- Mozilla method false positive alert rate at 12.5%, missed detection rate at 6.8%; ensemble methods reduce false alarms while maintaining or improving recall
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.18377.
Fig 1: Study Design Overview
Fig 2: Alert creation example
Fig 3: Simplified workflow of Mozilla’s alert creation process
Fig 4 (page 5).
Fig 5 (page 5).
Fig 6 (page 5).
Fig 5: Annotation user interface example
Fig 7: CPD agreement example: consensus threshold 𝐶of 2, tolerance margin M of 5
Limitations
- Annotations cover only 174 time series, which may limit generalizability across all Mozilla performance scenarios.
- Evaluation focuses on statistical CPD methods and does not incorporate machine learning or domain-specific heuristics that could complement.
- No adversarial robustness testing against deliberate manipulation or highly non-stationary performance degradation patterns.
- Analysis centers on two Mozilla test subsets (Speedometer3 and TP6), potentially limiting coverage of broader test configurations.
- Manual annotations, although multi-annotator, may still contain subjective labeling biases or disagreement on borderline cases.
- Integration and operational overhead of ensemble CPD methods in a continuous delivery environment was not fully quantified.
Open questions / follow-ons
- How do machine learning or domain-adapted methods compare or complement these general CPD techniques for performance anomaly detection?
- Can ensemble CPD strategies be tuned automatically per project or workload to optimize alert accuracy and minimize manual triage effort?
- What are the operational trade-offs (latency, compute cost) when deploying ensemble CPD methods in continuous integration systems at scale?
- How resilient are CPD methods and ensembles to adversarial or highly non-stationary performance degradation patterns introduced by complex software changes?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, this paper exemplifies the challenges of reliable anomaly detection in large-scale, noisy, continuous data environments where minimizing false positives and missed detections is critical to maintain end-user trust and operational efficiency. The comprehensive benchmarking of diverse CPD methods and ensemble strategies reveals that single statistical tests may not suffice, and ensembles can offer more robust detection.
The practitioner-grounded annotation approach and integration validation provide a useful model for rigorous evaluation of anomaly detection systems beyond naive statistical assumptions. Insights on precision-recall tradeoffs and operational integration considerations are directly applicable when designing detection mechanisms to identify automated abuse, suspicious activity, or other anomalous patterns in web traffic or user behavior. Furthermore, the release of standard benchmarking tools and annotated datasets sets a precedent for building common evaluation frameworks in bot detection and CAPTCHAs.
Cite
@article{arxiv2606_18377,
title={ Exploring Statistical Change Point Detection Techniques for Performance Anomaly Detection at Mozilla },
author={ Mohamed Bilel Besbes and Gregory Mierzwinski and Suhaib Mujahid and Philipp Leitner and Alexander Serebrenik and Dave Hunt and Diego Elias Costa },
journal={arXiv preprint arXiv:2606.18377},
year={ 2026 },
url={https://arxiv.org/abs/2606.18377}
}