Behaviour-Conditioned Neural Processes for Adaptive Residential Short-Term Load Forecasting

Source: arXiv:2607.16168 · Published 2026-07-17 · By Ramin Soleimani, Andrea Visentin, Dirk Pesch

TL;DR

Residential short-term load forecasting (STLF) is complex due to heterogeneity and temporal variability in household electricity consumption driven by diverse behavioral routines. Traditional forecasting models struggle to balance representational capacity with rapid adaptation to new data and behavioral heterogeneity. This paper introduces a behaviour-conditioned Attentive Neural Process (ANP) framework that integrates inferred behavioral structure directly into the probabilistic forecasting mechanism rather than treating behavior as an external grouping or auxiliary input. The key innovation is a dual latent representation: a discrete behavioral latent variable inferred from context conditions the decoder via feature-wise linear modulation (FiLM), while a continuous latent variable captures shared functional uncertainty over load profiles. A weakly supervised training approach uses clustering-derived labels as soft supervision, allowing test-time conditioning solely on inferred behavioral class distributions without access to ground-truth labels. Evaluation on the Smart Grid, Smart City (SGSC) residential dataset under user-disjoint splits, variable context lengths, and multiple forecast horizons shows consistent improvements in mean absolute error (MAE) and continuous ranked probability score (CRPS) relative to a baseline Attentive Neural Process and fixed-window deterministic baselines. Notably, the best FiLM-ANP variant reduces MAE by 7.9% and CRPS by 6.9% compared to ANP, with lower RMSE suggesting fewer large prediction errors on heterogeneous consumption patterns. This approach enables adaptive, uncertainty-aware STLF with a single global model that incorporates inferred behavioral structure internally for improved personalization and robustness.

Key findings

• FiLM-ANP-Soft variant reduces mean absolute error (MAE) by 7.9% on average compared to label-agnostic Attentive Neural Process (ANP) baseline across multiple forecast horizons and context sizes.
• Continuous ranked probability score (CRPS) decreases by 6.9% on average for FiLM-ANP-Soft vs ANP, indicating improved probabilistic forecast calibration.
• FiLM-ANP-Soft achieves lower root mean squared error (RMSE) than fixed-window deterministic STLF baselines at all evaluated horizons, indicating fewer large forecast deviations with heterogeneous user data.
• Behavioral latent variable conditioning yields largest gains under limited context lengths, showing improved adaptation to sparse historical data.
• Context-inferred behavioral class distributions enable label-free test-time behavioral conditioning without ground-truth labels or look-ahead data leakage.
• Dual latent variables explicitly separate global functional uncertainty (continuous) from behavior-specific variations (discrete), improving model capacity to capture heterogeneous residential load profiles.
• User-disjoint train/validation/test splits demonstrate model generalization to unseen households.
• Ablation studies confirm FiLM-based decoder conditioning outperforms label-agnostic ANP and fixed behavioral embeddings.

Methodology — deep read

The authors consider residential STLF as a context-conditioned function learning problem where each household load profile corresponds to a forecasting task. The model inputs a context set DC of observed input-output pairs (timestamps and loads) per profile and predicts target loads for future timestamps XT. They extend the Attentive Neural Process (ANP) framework by adding a discrete latent variable c representing behavioral modes and a continuous latent variable z capturing global functional uncertainty. These variables are modeled independently with context-conditioned priors p(c|DC) and p(z|DC) and target-conditioned posteriors q(c|DT) and q(z|DT). The discrete behavioral latent c modulates the Transformer-style decoder via FiLM (feature-wise linear modulation) layers to allow flexible adaptation to inferred consumption regimes. During training, weak supervision via clustering-derived behavioral labels is used to compute cross-entropy loss and a KL divergence regularization term aligning the label posterior with the context prior. A Gumbel-Softmax relaxation allows differentiable sampling of c. At test time, behavioral conditioning relies solely on the context-inferred prior p(c|DC), with no true labels available. The deterministic encoder embeds context points and target queries through cross-attention to produce target-specific embeddings r_t*, which combined with z and c condition the probabilistic decoder outputs parameterizing Gaussian predictive distributions. Training optimizes an evidence lower bound (ELBO) combining likelihood and KL terms with a temperature-controlled Gumbel-Softmax for discrete variables. They use the publicly available Smart Grid, Smart City (SGSC) dataset with user-disjoint train/validation/test splits, variable-length contexts (days/hours) for conditioning, and multi-step forecasts up to 48 steps. Performance is evaluated with MAE, CRPS, and RMSE compared to a label-agnostic ANP baseline and fixed-window deterministic models. An example end-to-end: Given a household's past N observations (context), the model computes context embeddings, infers behavioral class probabilities p(c|DC), samples z, applies FiLM modulation based on c to its Transformer decoder, and produces a distribution over future loads—capturing uncertainty and behavioral heterogeneity simultaneously. Code and pretrained model availability are not specified, so full reproducibility is uncertain.

Technical innovations

• Introduce a discrete latent behavioral variable inferred from context and used to condition the decoder in a Neural Process, enabling behavior-aware forecasting.
• Use dual latent variables—a continuous latent for global functional uncertainty and a discrete latent for behavior—to separate sources of variation within a single probabilistic framework.
• Apply feature-wise linear modulation (FiLM) in Transformer decoders for flexible, behavior-conditioned decoder adaptation.
• Employ weak supervision from clustering-derived labels during training to enable label-free behavior inference and conditioning at test time without look-ahead data.

Datasets

• Smart Grid, Smart City (SGSC) residential smart-meter dataset — size not explicitly specified — public
• Note: exact dataset size and splits not detailed in summary, likely thousands of user load profiles over multiple days/hours.

Baselines vs proposed

• Attentive Neural Process (ANP): Mean Absolute Error (MAE) = baseline; FiLM-ANP-Soft: MAE reduced by 7.9% relative to ANP
• ANP: Continuous Ranked Probability Score (CRPS) = baseline; FiLM-ANP-Soft: CRPS reduced by 6.9%
• Fixed-window deterministic STLF baseline: Root Mean Squared Error (RMSE) = higher; FiLM-ANP-Soft: lower RMSE across all forecast horizons

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.16168.

Fig 2: K-means K-selection criteria (inertia, silhouette, Davies–Bouldin, and Calinski–

Fig 3: Representative normalised load profiles given by K-means cluster centroids (52-

Fig 4: Percentage reduction in MAE and CRPS of FiLM-ANP–Soft relative to ANP

Fig 5: Relative improvement of FiLM-ANP–Soft over ANP as a function of context

Fig 5 (page 31).

Fig 6: Context-driven forecasting progression for the FiLM-ANP–Soft model under

Limitations

• The model's performance relies on quality of clustering-derived behavioral labels used as weak supervision; clustering imperfections could degrade latent behavior inference.
• The dataset and experiments focus on single-region SGSC data; generalization to other geographies or datasets is untested.
• No explicit adversarial or robustness evaluations are presented to test model behavior under anomalous or adversarial load patterns.
• Some hyperparameter choices, such as number of behavioral classes K, Gumbel-Softmax temperature, or latent dimensions, are not extensively explored.
• Reproducibility is limited by absence of publicly available code, pretrained weights, or detailed training recipes.
• The impact of distribution shifts over longer time frames or seasonal effects beyond the available context is not analyzed.

Open questions / follow-ons

• How does the number of behavioral latent classes K influence forecasting accuracy and adaptability?
• Can the approach generalize across different geographical regions or incorporate spatial correlations alongside behavioral conditioning?
• How robust is the model to sudden changes in user behavior or anomalies (e.g., outages, holidays) that may break behavioral assumptions?
• What are the trade-offs between behavioral conditioning via discrete latent variables versus continuous embeddings or other conditioning mechanisms?

Why it matters for bot defense

From a bot-defense or CAPTCHA-related perspective, this work illustrates an advanced conditioning approach that integrates inferred latent behavioral structure directly into a neural forecasting model via latent variables and FiLM modulation. This strategy highlights the benefit of explicitly modeling heterogeneous behavioral regimes within a unified probabilistic prediction model rather than relying on external grouping or fixed partitions. Bot-defense engineers could draw parallels in user-behavior modeling: embedding inferred behavior classes internally within adaptive, uncertainty-aware models could improve detection or prediction of user interactions under diverse and evolving behavioral regimes. The weak supervision technique to infer discrete latent classes without labeled data is also relevant, suggesting potential for leveraging clustering or unsupervised signals during training while relying solely on inferred context at run-time. However, the domain (energy forecasting) and continuous prediction task differ substantially from bot-interaction detection. The paper's methodology could inspire adaptive feature modulation conditioned on inferred user types or browsing habits to enhance resilience to behaviorally diverse bots or fuzzier attack patterns, especially under limited observed context. Overall, the approach encourages unified adaptive modeling of latent behavior to improve accuracy and robustness—a principle applicable across security and CAPTCHA domains when behavioral heterogeneity is significant.

Cite

@article{arxiv2607_16168,
  title={ Behaviour-Conditioned Neural Processes for Adaptive Residential Short-Term Load Forecasting },
  author={ Ramin Soleimani and Andrea Visentin and Dirk Pesch },
  journal={arXiv preprint arXiv:2607.16168},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.16168}
}

Read the full paper

• arXiv:2607.16168 (abstract)
• arXiv:2607.16168 (PDF)