Extreme Adaptive Transformer for Time Series Forecasting
Source: arXiv:2607.02437 · Published 2026-07-02 · By Sanjeev Shrestha, Hui Liu, Yifan Zhang
TL;DR
This paper addresses the challenge of accurately forecasting time series that contain rare but crucial extreme events, with a focus on hydrologic streamflow forecasting where extreme peaks are impactful for flood monitoring and water resource management. Standard Transformer-based forecasting models, while effective in capturing long-range dependencies, tend to treat all time points uniformly and thus may underrepresent important extreme events due to their rarity. The authors propose the Extreme-Adaptive Transformer (Exformer), a novel Transformer architecture that introduces an extreme-adaptive attention mechanism consisting of Local, Stride, and Extreme components. This mechanism adaptively adjusts query-key interactions based on whether the tokens correspond to normal or extreme events, preserving important extreme-to-extreme dependencies while efficiently modeling normal temporal patterns. Experiments on four real-world hydrologic datasets demonstrate that Exformer outperforms multiple state-of-the-art baselines, achieving superior 3-day streamflow forecasting accuracy (lower RMSE and MAPE) while also significantly reducing attention computation cost compared to full attention and other Transformer variants. Ablations confirm the importance of each attention component, especially the Extreme attention.
Key findings
- Exformer achieves best 3-day forecast accuracy on 7 of 8 RMSE and MAPE metrics across four hydrologic datasets (Ross, Saratoga, UpperPen, SFC).
- Exformer reduces MAPE by 50%, 20%, 33.3%, and 33.3% on Ross, Saratoga, UpperPen, and SFC respectively compared to the closest Transformer baseline PFformer.
- Compared to DAN, an extreme-adaptive baseline, Exformer reduces RMSE on all datasets and MAPE by up to 73.3% (UpperPen).
- Exformer lowers RMSE by between 7.9% and 31.6% relative to iTransformer across datasets.
- Extreme-adaptive attention reduces self-attention complexity to O((w + s)I/p + ne^2), where w,s (local window, stride) and p (patch size) are small constants and ne (extreme token count) is very small, enabling scalable long-sequence forecasting.
- Exformer requires only 31.6 GFLOPs, 14.3M parameters, and 241MB GPU memory versus 187 GFLOPs and 705MB memory for PFformer, demonstrating over 5x computational efficiency gains.
- Ablation of Local, Stride, and Extreme attention components shows combined use yields best overall accuracy, with Extreme component alone close to full model performance.
- Extreme-Adaptive Attention outperforms standard Dozer, canonical full attention, AutoCorr, FedAttn, and ProbSparse attention on RMSE and MAPE in 3 of 4 datasets.
Threat model
The adversary is an implicit forecasting environment facing highly imbalanced time series with rare extreme events that are critical to predict accurately. There is no active adversary attempting to corrupt data or model; instead, the challenge lies in the model's ability to detect and emphasize rare extreme temporal patterns against a majority of normal observations. The model cannot rely on large volumes of extreme event data since they are inherently scarce, so its architecture must preserve rare-to-rare dependencies while ignoring irrelevant normal-to-extreme interactions.
Methodology — deep read
The authors target the problem of forecasting multivariate time series with rare extreme events, specifically hydrologic streamflow affected by floods. The threat model assumes a forecasting environment with imbalanced data where extreme events are scarce and possibly underrepresented by standard uniform attention mechanisms.
Data: Experiments use four real-world streamflow and rainfall multivariate datasets from Santa Clara County, CA (Ross, Saratoga, UpperPen, SFC). Each dataset contains 15-minute interval recordings spanning January 1988–May 2022. Training and validation samples are drawn randomly from 1988 to August 2021; the test set corresponds to the wet season from September 2021 to May 2022. Data is preprocessed by log-transform (log(x+1)) and standardization. Extreme events are labeled using a Gaussian Mixture Model-derived outlier score with a chosen threshold.
Architecture: Exformer is an encoder-only Transformer. Inputs are decomposed into seasonal and trend components using prior decomposition methods. Seasonal components are fed into the Exformer encoder; trend components are linearly predicted. The encoder input is patchified along the temporal axis—creating non-overlapping patches—and each patch is assigned a normal/extreme label.
The core novelty is the Extreme-Adaptive Attention mechanism combining three sparse attention components: Local attention attends nearby normal patches within a window; Stride attention attends normal patches at fixed periodic intervals; Extreme attention allows extreme token queries to attend only extreme token keys. Attention masks integrate a Dozer-style sparse mask (local + stride) with an extreme-aware mask derived from patch labels. Normal queries use masked local + stride attention only on normal tokens; extreme queries attend exclusively to extreme tokens, capturing rare extreme-to-extreme temporal dependencies while reducing irrelevant computations.
Training used standard Transformer training regimes but exact epochs and hyperparameters are not explicitly detailed in the text provided. The forecasting task is to predict streamflow for 3 days ahead (288 steps at 15-minute intervals), i.e., long-term forecasting.
Evaluation: Performance metrics are root mean squared error (RMSE) and mean absolute percentage error (MAPE). Baselines include several state-of-the-art linear methods (NLinear, DLinear), Transformer variants (Informer, FEDformer, PFformer, iTransformer), extreme-event methods (DAN, NEC+), and LSTM variants. Ablation studies isolate each attention component's contribution. Attention mechanism comparisons replace the extreme-adaptive attention with standard Dozer, canonical full attention, AutoCorrelation, FedAttn, and ProbSparse to confirm its advantages.
Reproducibility: Code is publicly released on GitHub at https://github.com/sanzexstha/Exformer. The datasets appear public or accessible via prior studies. No frozen weights or further replication details are provided.
Example end-to-end: Given a historical multivariate input sequence (e.g., streamflow and rainfall), the model first decomposes the input into seasonal and trend components. The seasonal time series is patchified and embedded into tokens, each labeled normal or extreme via an outlier score threshold. During self-attention, normal tokens use sparse attention focusing on local and stride neighbors also labeled normal, while extreme tokens attend only to other extreme tokens to emphasize rare event patterns. The encoded seasonal output and the predicted trend are combined to generate the final forecast for the next 3 days. The model is trained and validated on historical data and tested on a held-out wet period with extreme hydrologic events. Forecast accuracy (RMSE/MAPE) and computational efficiency (FLOPs, memory) are evaluated and compared with baselines.
Technical innovations
- Extreme-Adaptive Attention mechanism dynamically adjusts sparse attention patterns conditional on token extremity labels, enabling selective extreme-to-extreme attention and normal local/periodic attention.
- Integration of Local, Stride, and Extreme sparse attention components to separately capture short-range, periodic, and rare extreme dependencies within a single Transformer encoder.
- Efficient patch-based temporal tokenization combined with outlier detection labeling to create fine-grained extreme-aware attention masks.
- A hybrid forecasting architecture that decomposes time series into seasonal and trend components, using extreme-adaptive attention only on seasonal patterns for improved extreme event modeling.
Datasets
- Ross — Streamflow and rainfall hydrologic data from Santa Clara County, CA — Publicly sourced from Li & Anastasiu (2025)
- Saratoga — Streamflow and rainfall hydrologic data from Santa Clara County, CA — Publicly sourced from Li & Anastasiu (2025)
- UpperPen — Streamflow and rainfall hydrologic data from Santa Clara County, CA — Publicly sourced from Li & Anastasiu (2025)
- SFC — Streamflow and rainfall hydrologic data from Santa Clara County, CA — Publicly sourced from Li & Anastasiu (2025)
Baselines vs proposed
- PFformer: RMSE = 4.21 (Ross) vs Exformer = 4.20; MAPE = 0.10 (Ross) vs Exformer = 0.05
- DAN: RMSE = 4.25 (Ross) vs Exformer = 4.20; MAPE = 0.07 (Ross) vs Exformer = 0.05
- iTransformer: RMSE = 4.56 (Ross) vs Exformer = 4.20; MAPE = 0.57 (Ross) vs Exformer = 0.05
- FEDformer: RMSE = 6.01 (Ross) vs Exformer = 4.20; MAPE = 2.10 (Ross) vs Exformer = 0.05
- Informer: RMSE = 7.84 (Ross) vs Exformer = 4.20; MAPE = 4.05 (Ross) vs Exformer = 0.05
- Extreme-Adaptive Attention: RMSE = 4.20 (Ross) vs Dozer Attention = 4.20; MAPE = 0.05 (Ross) vs Dozer = 0.08
- Extreme-Adaptive Attention: RMSE = 1.61 (Saratoga) vs Dozer = 1.70; MAPE = 0.07 (Saratoga) vs Dozer = 0.09
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.02437.
Fig 3: visualizes the forecasting results on Saratoga at horizon 288 by comparing the predicted
Fig 4: Sensitivity analysis of 3-day prediction across different threshold values
Fig 3 (page 11).
Fig 4 (page 11).
Limitations
- The model and experiments target hydrologic datasets with two variables (streamflow, rainfall); generalizability to higher-dimensional or different domains is untested.
- The threshold for extreme event labeling depends on outlier detection hyperparameters (GMM threshold), which may affect attention masking and is not extensively analyzed.
- No adversarial or robustness evaluation under manipulated extreme event patterns or corrupted data.
- The training regime details such as epochs, optimizer choice, learning rate schedule, random seed controls are not fully disclosed, limiting reproducibility clarity.
- Evaluation focuses on 3-day ahead forecasting; performance on very long horizons or across seasons with different extreme event frequencies is not reported.
- Dependency on prior decomposition into seasonal and trend components may limit applicability to datasets where such decomposition is noisy or unavailable.
Open questions / follow-ons
- How sensitive is the model's performance to the threshold and method used for labeling extreme events?
- Can the Extreme-Adaptive Attention mechanism be generalized to multivariate time series with more variables or other domains beyond hydrology?
- How does the model perform under distribution shifts or in forecasting unseen types of extreme events?
- Can the approach be integrated with uncertainty quantification or probabilistic forecasting to better capture extreme event risks?
Why it matters for bot defense
From a bot-defense and CAPTCHA learning analyst perspective, this work is valuable as it advances techniques for detecting and emphasizing rare but critical events in time series data via adaptive attention mechanisms. Such adaptive attention strategies could inform architectural designs in anomaly detection or behavioral biometrics where extreme but sparse signals must be modeled effectively. The efficient sparse attention approach also demonstrates a practical balance of accuracy and compute cost, relevant to deploying frontend defensive models with latency constraints. However, this paper does not directly address adversarial robustness or security-specific threat models common in bot defense. It primarily contributes to modeling rare event temporal dependencies with computational efficiency, which could inspire novel defense features that detect subtleties in user interaction time patterns or traffic spikes.
Cite
@article{arxiv2607_02437,
title={ Extreme Adaptive Transformer for Time Series Forecasting },
author={ Sanjeev Shrestha and Hui Liu and Yifan Zhang },
journal={arXiv preprint arXiv:2607.02437},
year={ 2026 },
url={https://arxiv.org/abs/2607.02437}
}