SEP-PRISM Data: A multi-source dataset for solar energetic particle forecasting

Source: arXiv:2607.16160 · Published 2026-07-17 · By Yian Yu, Yang Chen, Lulu Zhao, Kathryn Whitman, Ward Manchester, Tamas Gombosi

TL;DR

Solar energetic particle (SEP) events pose significant space weather hazards impacting astronauts, spacecraft, and high-latitude technology. Accurate 24-hour ahead forecasting of operational SEP events (proton flux > 10 pfu at >10 MeV) remains challenging due to heterogeneous observational data varying in cadence, coverage, and format. This paper presents SEP-PRISM Data, a carefully curated multi-source, long-range dataset integrating flare records, active-region magnetic field parameters, coronal mass ejection (CME) catalogs, GOES soft X-ray flux, and proton flux observations spanning 1986 to 2025. By aligning and fusing sources including SHARP/SMARP magnetic products and DONKI/CDAW CME records into unified archives (SMHARP and CDAWDONKI), the authors produce a consistent, window-based dataset summarized in non-overlapping 24-hour intervals with statistical aggregations of predictors paired with labels for the subsequent 24-hour period. The resulting dataset includes 14,464 samples with 650 operational SEP positives, suitable for reproducible benchmarking and feature analysis. Validation shows strong alignment between magnetic and CME catalogs, preserving long-term temporal coverage without truncation. The SEP-PRISM Data enables uniform supervised learning experiments for SEP forecasting and facilitates future advances in operational space weather prediction.

Key findings

• SEP-PRISM Data integrates five heterogeneous sources into a unified, supervised learning dataset with 14,464 labeled 24-hour samples, including 650 operational SEP events (≥10 pfu proton flux).
• Magnetic predictors from SHARP (2010–2025) and SMARP (1996–2010) were aligned via regression, improving Pearson correlation coefficients from ~0.77 to 0.90 for key parameters such as R_VALUE and USFLUXL (e.g., USFLUXL COR improved from 0.9605 to 0.9849).
• CME records from DONKI (2010–2025) and CDAW (1996–2025) were fused into the CDAWDONKI archive, increasing speed correlation from 0.5562 to 0.6479 and decreasing RMSE between catalogs.
• The dataset covers February 3, 1986 through September 10, 2025, supporting long-term validation and operational forecasting over multiple solar cycles.
• Predictors are aggregated over non-overlapping 24-hour historical windows using minimum, mean, and maximum statistics, paired with labels over the subsequent 24-hour window for SEP event occurrence and auxiliary regression targets such as flare and CME counts.
• Missing or invalid data was handled by nearest-neighbor imputation within active regions for magnetic data and k-nearest neighbor imputation for CME event features.
• Derived magnetic field parameters include unsigned flux, mean inclination, current density, twist, shear metrics, and size descriptors, with regression used to reconstruct SHARP-only variables from SMARP inputs pre-2010.
• Operational SEP events constitute ~4.5% of samples (650/14,464), illustrating rarity and class imbalance challenges for forecasting models.

Threat model

The adversary scenario is implicit: the forecasting system is tasked with predicting SEP events 24 hours ahead based on observed solar and heliospheric drivers. The adversary could be natural solar variability and measurement noise causing forecasting errors. There is no indication of intentional attacks or manipulation capabilities, and data fusion assumes benign measurement errors and catalog inconsistencies as the primary 'threat' to data quality rather than an active adversary.

Methodology — deep read

The authors start with the threat model of operational SEP forecasting, where the goal is to predict proton flux exceeding 10 pfu in the GOES > 10 MeV channel 24 hours into the future. Data collected range from solar flare catalogs, photospheric magnetic field products, CME catalogs, GOES soft X-ray flux, and proton flux time series. The SEP event labels come from the CLEAR SEP benchmark dataset spanning 1986 to 2025.

Data provenance includes SHARP vector magnetic field parameters from SDO/HMI (2010-2025), extended backward to 1996 using aligned SMARP data from SOHO/MDI. Similarly, CME catalogs from DONKI (2010-2025) and CDAW (1996-2025) are fused for longer temporal coverage. Flare data and GOES satellite measurements of proton flux and X-ray flux span the full 1986–2025 period.

Key preprocessing steps include (1) treating missing or invalid magnetic measurements via time-based nearest neighbor imputation within each active region, (2) regression-based alignment between SMARP and SHARP parameters to generate a unified SMHARP magnetic dataset covering 1996 onward, including reconstruction of SHARP-only magnetic parameters from SMARP inputs, (3) aligning CME events between DONKI and CDAW in the overlapping period using time proximity and regression to produce the CDAWDONKI unified CME archive, excluding poorly reconstructed latitude, (4) binning all heterogeneous inputs into fixed, non-overlapping 24-hour windows, computing three summary statistics per predictor (minimum, mean, maximum), and pairing these with future 24-hour window targets representing SEP event occurrence and auxiliary regression targets such as flare counts and CME speeds.

The resulting tabular dataset contains 274 columns representing summarized features aggregated over the past 24 hours and labeled targets over the subsequent 24 hours.

The training regime is not exhaustively described here; however, companion works use SEPNET-PRISM predictive models trained on this dataset. The authors validate fusion quality through correlation and RMSE statistics on temporally overlapping windows, showing improved coherence between merged datasets.

Evaluation uses Pearson correlations and RMSE to quantify agreement between aligned SMARP-SHARP and DONKI-CDAW features. The dataset includes provisions to identify imputed versus observed data for provenance checks. Missing data strategies and window-based aggregation ensure uniformity and comparability across samples spanning four decades.

Reproducibility is achieved through published CSV datasets at three tiers (raw, processed, analyzed), processing scripts in notebooks and R scripts for fusion, alignment, and aggregation. The entire dataset and pipeline are documented with timestamped records in UTC, facilitating consistent experimental setups. Code for dataset refresh and near-real-time update is provided in the repository. However, no pretrained forecasting model weights are included, focusing on data availability.

A concrete example: To form a training sample for, say, June 1, 2010, the dataset aggregates all flare, CME, magnetic, proton, and X-ray measurements during June 1 (00:00-24:00 UTC), computes minimum, mean, max for each predictor, then assigns labels indicating whether an operational SEP event occurred from June 2 (00:00-24:00 UTC). This enables supervised classification or regression for 24-hour ahead SEP forecasting.

Thus, SEP-PRISM Data is structured to address historic heterogeneity and gaps in solar datasets by using regression-based reconstructions, harmonized multi-source fusion, and uniform windowing to deliver a consistent machine learning benchmark resource spanning multiple solar cycles.

Technical innovations

• Creation of a unified SMHARP magnetic archive by aligning and reconstructing SHARP magnetic parameters from SMARP observations via regression models to extend temporal coverage back to 1996.
• Fusion of DONKI and CDAW CME catalogs into a synchronized CDAWDONKI archive with regression-based alignment to address catalog inconsistencies and extend CME data coverage.
• A window-based supervised learning dataset engineering approach aggregating heterogeneous solar driver observables into uniform fixed 24-hour summaries with min, mean, max statistics paired with future 24-hour prediction targets.
• Provision of provenance flags to allow filtering or stratification of fused versus original data enabling rigorous downstream analysis accounting for data quality and source.
• Publicly released multi-tier dataset (raw, processed, analyzed) and fully documented reproducible processing pipeline scripts facilitating transparent benchmarking and operational SEP forecasting research.

Datasets

• CLEAR SEP Benchmark — 569 events spanning 1986–2025 — publicly available compiled SEP event catalog
• SHARP — ~4.9M active region observations with vector magnetic parameters — SDO/HMI from 2010–2025
• SMARP — ~68K active region magnetic observations — SOHO/MDI from 1996–2010
• DONKI CME catalog — 8,366 CME events from 2010–2025 — NASA's DONKI database
• CDAW CME catalog — 16,216 CME events from 1996–2025 — publicly available CME catalog
• GOES soft X-ray flux — ~4.1M 5-minute cadence measurements from 1986–2025 — NOAA GOES satellites
• GOES proton flux >10 MeV — ~4.1M 5-minute cadence measurements from 1986–2025 — NOAA GOES satellites

Baselines vs proposed

• SHARP-SMARP magnetic parameter fusion: USFLUXL Pearson correlation improved from 0.9605 to 0.9849 after regression-based alignment
• SHARP-SMARP magnetic parameter fusion: MEANGBL correlation improved from 0.7689 to 0.8433 with RMSE reduced from 30.19 to 2.55
• DONKI-CDAW CME speed parameter correlation improved from 0.5562 to 0.6479 after regression alignment with reduced RMSE
• Operational SEP events constitute 4.5% of samples (650/14,464), establishing a relevant but imbalanced forecasting target

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.16160.

Fig 1: Timeline of multi-source data used in this study from 3 February 1986 to 10 September 2025 (adapted from a figure

Fig 2: Comparison of SMARP observations, SHARP observations, and regression-based fitted SHARP values for three

Limitations

• Despite improved alignment, some magnetic parameters related to helicity and vector derivatives exhibit moderate or low correlations (≈0.44–0.50), limiting feature fidelity.
• CME latitude parameter alignment between DONKI and CDAW showed poor correlation (0.0821) and high RMSE, resulting in its omission from the final feature set.
• The dataset is inherently imbalanced due to the rarity of operational SEP events (4.5% positive), posing challenges for balanced predictive modeling.
• No adversarial or adversary-aware evaluations are presented to assess dataset robustness against manipulated or noisy inputs.
• Temporal aggregation into 24-hour windows may lose finer temporal dynamics relevant for rapid SEP onset forecasting.
• While data fusion aims to extend coverage, reconstructed surrogate data (e.g., SMARP-derived SHARP parameters) are approximations that may introduce errors or biases.

Open questions / follow-ons

• How can the aligned surrogate magnetic and CME parameters be further improved to better capture complex solar magnetic vector fields and CME geometry?
• Can multi-task or hierarchical models leveraging the auxiliary regression targets (flare counts, CME speeds) improve operational SEP event forecasting given the rarity of positives?
• What approaches to mitigate class imbalance and temporal dynamics beyond fixed 24-hour windows would boost forecasting performance?
• How does the SEP-PRISM Data perform under distribution shifts during extreme solar events or new solar cycles post-2025?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, SEP-PRISM Data exemplifies the challenges and solutions in integrating heterogeneous, multi-source time series data with different cadences, coverage periods, and noise characteristics into a unified machine learning-ready format. The regression-based alignment approach to fuse related but discordant scientific catalogs, provision of provenance metadata, and window-based statistical summarization offer relevant strategies for feature engineering in security domains where multi-sensor or multi-source telemetry must be aligned and aggregated under strict temporal constraints. Furthermore, the handling of sparse positive event labeling and severe class imbalance encountered in SEP event forecasting parallels challenges in anomaly or bot detection.

The curated dataset and processing pipeline demonstrate best practices in reproducible data fusion and creation of labeled supervised learning benchmarks over multiple decades of observational data. Similar rigor and documentation could benefit bot-defense datasets where variability in attacker behavior and measurement schemes complicate model training and evaluation. Finally, the detailed validation metrics (correlations, RMSEs) on aligned features underscore the importance of assessing fusion quality to avoid artifact-driven model outcomes—critical for robust security system design.

Cite

@article{arxiv2607_16160,
  title={ SEP-PRISM Data: A multi-source dataset for solar energetic particle forecasting },
  author={ Yian Yu and Yang Chen and Lulu Zhao and Kathryn Whitman and Ward Manchester and Tamas Gombosi },
  journal={arXiv preprint arXiv:2607.16160},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.16160}
}

Read the full paper

• arXiv:2607.16160 (abstract)
• arXiv:2607.16160 (PDF)