How Good Can Linear Models Be for Time-Series Forecasting?

Source: arXiv:2606.27282 · Published 2026-06-25 · By Lang Huang, Jinglue Xu, Luke Darlow

TL;DR

This paper revisits the conventional wisdom that scaling model capacity is necessary for improved long-term time series forecasting. Instead, it demonstrates that a well-tuned linear model, specifically Ridge regression with closed-form solutions, can close much of the accuracy gap at far lower computational cost by carefully optimizing preprocessing hyperparameters. The authors perform extensive per-series and per-horizon hyperparameter searches over lookback window size, local normalization scopes, regularization strength, and data augmentation on eight standard benchmarks. Surprisingly, they find the optimal lookback-horizon relationship is highly dataset-specific and often non-monotonic, local normalization on a trailing fraction of the window outperforms normalization over the entire context, and series within the same dataset can have very different optimal hyperparameters. Their tuned linear models outperform prior linear models and surpass Transformer, MLP, and CNN baselines on six of eight datasets. The analysis also reveals that optimized hyperparameters provide valuable diagnostic insights into data structure that black-box deep models implicitly absorb. A publicly released SearchCast pipeline enables reproduction and further dataset diagnosis.

Key findings

• Optimized Ridge regression reduces MSE by up to 14.8% over OLS and prior linear forecasters (e.g., 4.8% on ETTm2, 4.3% on ETTh1, 14.8% on Exchange).
• The optimal lookback horizon scaling exponent b ranges widely, from +0.46 on ETTm2 to -0.19 on Exchange and Traffic.
• Local normalization on a trailing fraction r of the lookback window (r between 0.3% and 30%) is almost always preferred over full-window normalization (r=1).
• Series-level variation within datasets is substantial: some datasets (e.g., Weather) benefit from fully per-series hyperparameter tuning, while others (ETT benchmarks) perform best with fully shared hyperparameters.
• Series grouping strategies for hyperparameter sharing reveal dataset-specific optimal degrees of sharing, e.g., gs=1 (per-series) best for Weather, gs=21 degrades MSE by 10%.
• Horizon grouping (sharing hyperparameters across forecast horizons) incurs minimal MSE degradation (≤0.4%) for neighborhood sizes up to 48 forecast steps.
• The optimized linear model matches or beats nonlinear Transformer, MLP, and CNN baselines on 6 of 8 benchmarks, losing only on large multi-channel datasets Electricity and Traffic where shared representations may help more complex models.
• Augmentation with Gaussian noise in either time or frequency domains improves results and is selected in about 60-70% of trials across datasets.

Threat model

n/a — The paper addresses supervised forecasting accuracy rather than adversarial threat scenarios or attacks.

Methodology — deep read

The threat model is standard supervised forecasting with clean historical multivariate time series; adversarial attacks are not addressed. The main focus is on hyperparameter optimization rather than model robustness.

Data comes from eight widely used multivariate forecasting benchmarks spanning domains such as electricity (ETT variants), weather, traffic, exchange rates, and electricity consumption. Dataset splits are chronological using standard 60/20/20 or 70/10/20 train/val/test splits with temporal order preserved. Input channels range from 8 to 862 variates with forecast horizons up to 720 steps.

The model is Ridge regression with a closed-form solution W* = YX^T(XX^T + αI)^-1, where the weight matrix W maps a lookback window x of length L to forecast horizon H outputs. Regularization α is chosen via inner grid search. The inputs are univariate series treated independently (channel-independent), consistent with prior linear and Transformer baselines.

The core novelty lies in the extensive hyperparameter search pipeline SearchCast, which optimizes four preprocessing axes per forecast horizon slice and per variate group: (1) context lookback length L searched logarithmically over [32, 2048]; (2) normalization scope—global or local—and local trailing fraction r searched on a log scale (0.001 to 1); standardization or robust median/IQR normalization; (3) data augmentation with Gaussian noise either in time or frequency domain, or none, with noise intensity σ searched logarithmically from 0.001 to 0.5; (4) regularization α searched on a logarithmic grid [1e-6, 1e3].

To reduce combinatorial explosion, they cluster forecast horizons into groups of size gh and series into groups gs to share hyperparameters, sweeping from fully global to fully local. Cross-validation using 3-fold expanding window respects temporal ordering during search and evaluation. The best hyperparameters per group applied to held-out test sets.

Training is instantaneous on GPU due to closed-form Ridge solution, enabling large-scale Optuna Bayesian hyperparameter optimization with 20 trials per group cell. Results are reported on per-dataset, per-horizon, and per-series levels.

Evaluation primarily uses mean squared error (MSE) compared to strong recent baselines including linear (OLS, FITS, DLinear) and nonlinear (PatchTST, iTransformer, TimeMixer, TimesNet, Autoformer) methods.

The authors visualize per-horizon optimal lookbacks showing dataset-dependent divergence from monotonic relationships, local normalization ratios, augmentation usage, and weight heatmaps across lags. They perform ablations on grouping strategies to characterize cross-series and horizon hyperparameter sharing.

Reproducibility is supported by releasing the SearchCast pipeline for standardized dataset-specific hyperparameter diagnostics and search; however, some datasets are proprietary and full code for baselines is not included with the paper.

Technical innovations

• Demonstrating that extensive tuning of preprocessing hyperparameters (context length, local normalization ratio, augmentation, regularization) in Ridge regression can close much of the accuracy gap with larger nonlinear models for time series forecasting.
• Introducing a grouped hyperparameter search scheme over both forecast horizons and series groups that interpolates between fully global and fully local tuning, enabling tractable search in high dimensions.
• Revealing that optimal context lookback length scales non-monotonically with forecast horizon in a dataset-specific manner, challenging common assumptions in the literature.
• Showing the benefit of local normalization over a learned trailing fraction of the context window rather than normalization over the entire lookback, improving model adaptivity to local nonstationarity.
• Using hyperparameter search results as an interpretable diagnostic lens into dataset structure and series heterogeneity, exposing phenomena typically absorbed silently by deep models.

Datasets

• ETTh1 — hourly temperature transformer data — public benchmark
• ETTh2 — hourly temperature transformer data — public benchmark
• ETTm1 — 15-min temperature transformer data — public benchmark
• ETTm2 — 15-min temperature transformer data — public benchmark
• Weather — 21 meteorological indicators at 10-min intervals — public benchmark
• Electricity — consumption data from 321 clients hourly — public benchmark
• Traffic — hourly readings from 862 road sensors — public benchmark
• Exchange — daily exchange rates of 8 countries — public benchmark

Baselines vs proposed

• OLS linear: MSE = 0.395 avg on Exchange vs SearchCast Ridge: 0.341 (14.8% reduction)
• FITS linear: MSE = 0.400 avg on Exchange vs SearchCast: 0.341 (14.8% reduction)
• DLinear linear: MSE = 0.395 avg on Exchange vs SearchCast: 0.341
• PatchTST (transformer): MSE = 0.366 avg on Exchange vs SearchCast: 0.341
• SearchCast linear outperforms nonlinear PatchTST on 6 of 8 datasets (exceptions Electricity tied at 0.159 and Traffic 0.404 vs 0.391)
• Against nonlinear MLP and CNN baselines (TimeMixer, TimesNet, Autoformer), SearchCast achieves best MSE on majority benchmarks

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.27282.

Fig 1: Per-horizon optimal context-horizon relationships for four time series. The context lengths

Fig 2: Left: Median optimal lookback per (series, horizon) varies. Right: Adapting context per

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5: Forecast comparison. Our method (blue) closely tracks the ground truth (black), while the

Fig 6: Weight magnitude |w| over lag and forecast horizon. Lighter shades indicate larger

Fig 7: Per-series hyperparameter (local ratio r and regularization α) heatmaps, with gh = 48.

Fig 8: Augmentation analysis. Left: Proportion of trials selecting frequency-domain noise, time-

Limitations

• Focuses on classical supervised forecasting; no adversarial or robustness evaluation.
• Hyperparameter search can be computationally intensive despite efficiency of Ridge regression.
• Datasets are limited to widely used benchmarks without exploring more complex or irregular time series.
• Series-level optimal hyperparameters require large search budgets; scalability to even larger datasets is unclear.
• Study does not evaluate impact of missing data or real-world noisy conditions.
• Only considers independent channel modeling; interactions between channels not leveraged unlike some transformer architectures.

Open questions / follow-ons

• Can these preprocessing tuning insights extend to nonlinear deep forecasting models to improve their sample efficiency?
• How does series heterogeneity and partial hyperparameter sharing affect transfer learning or multi-task forecasting?
• What is the impact of missing data, irregular sampling, or noisy inputs on the optimal preprocessing configurations?
• Can the diagnostic hyperparameter patterns discovered inform the design of new adaptive normalization or data augmentation layers in deep models?

Why it matters for bot defense

Although this work is not directly related to bot defense or CAPTCHA system design, it offers valuable methodological lessons for bot-defense engineers deploying time series forecasting for traffic or user behavior prediction. The key takeaway is that careful preprocessing and hyperparameter tuning, rather than blindly scaling model capacity, can yield large accuracy improvements at much lower computational cost. The finding that normalization should focus on recent history rather than the entire context to adapt to nonstationarity may inform feature standardization in behavioral data based bot detection. The approach of grouped hyperparameter search balancing global and per-series tuning could inspire more efficient domain adaptation or user-segmentation methods in fraud or abuse detection pipelines. Finally, the use of interpretable linear models to diagnose data characteristics is a useful complement to opaque deep systems commonly used in CAPTCHA and bot defense, enabling more principled troubleshooting and trust.

Cite

@article{arxiv2606_27282,
  title={ How Good Can Linear Models Be for Time-Series Forecasting? },
  author={ Lang Huang and Jinglue Xu and Luke Darlow },
  journal={arXiv preprint arXiv:2606.27282},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.27282}
}

Read the full paper

• arXiv:2606.27282 (abstract)
• arXiv:2606.27282 (PDF)