A Two-Step Ensemble Score Filter for Data Assimilation in Partially Observed Systems

Source: arXiv:2606.28264 · Published 2026-06-26 · By Zixiang Xiong, Feng Bao, Hristo G. Chipilski, Siming Liang, Jingqiao Tang, Guannan Zhang

TL;DR

This paper addresses the challenge of data assimilation (DA) in partially observed, nonlinear, chaotic systems where only a subset of the state variables is directly observed. Standard Ensemble Score Filters (EnSF), which leverage nonlinear score-based corrections without requiring training, struggle to propagate updates effectively from observed to unobserved variables under sparse observation networks. To improve this, the authors propose a hybrid two-step method called Ensemble Score Filter with Linear Regression (EnSF-LR). EnSF-LR first applies the nonlinear EnSF update to observed state components, then uses an ensemble-based covariance regression (as in Ensemble Kalman Filters, EnKFs) to map the observed increments onto unobserved components. This approach combines the strength of nonlinear score-based filtering with the proven covariance-informed regression of EnKF for unobserved states.

The authors empirically evaluate EnSF-LR on the Lorenz-63 and Lorenz-96 model systems with both linear and nonlinear observation operators and sparse observation networks. Results show that for linear observations, EnSF-LR substantially reduces errors in the unobserved components compared to the original EnSF, approaching the accuracy of EnKF. For nonlinear observations, EnSF-LR outperforms both EnSF and EnKF in full-state estimation accuracy, demonstrating the advantages of hybridizing score-based nonlinear updates with linear regression for partially observed nonlinear filtering problems. This study presents an effective strategy to tackle sparse and nonlinear observations in sequential data assimilation by leveraging ensemble statistics.

Overall, the work advances ensemble filtering methodology for complex nonlinear dynamical systems by integrating generative AI-inspired score filtering with classical covariance regression. The proposed EnSF-LR method improves propagation of observational information to unobserved variables, yielding superior state estimation accuracy under challenging partial and nonlinear observation scenarios.

Key findings

• EnSF-LR significantly reduces full-state root-mean-square error (RMSE) compared to original EnSF on both Lorenz-63 and Lorenz-96 with sparse linear observations (Fig. 1).
• Under linear observations, EnSF-LR moves EnSF accuracy close to stochastic EnKF baseline but does not fully reach EnKF performance.
• For nonlinear (arctan) observation operators, EnSF-LR outperforms both EnSF and EnKF in full-state RMSE, e.g., Lorenz-63 mean RMSE reduced from ~6.3 (EnSF) and ~4.2 (EnKF) to ~2.7 (EnSF-LR) (Table 3).
• The linear regression step projects observed-state analysis increments to unobserved components using prior cross-covariance (Eq. 13), improving unobserved variable correction.
• EnSF alone updates only observed components due to the zero likelihood gradient in unobserved subspace (Eq. 8), causing large unobserved errors under sparse observations.
• Large ensembles (Ne=1000 for Lorenz-63, Ne=10,000 for Lorenz-96) were used to reduce sampling error and isolate filter approximation error.
• The two-step EnSF-LR scheme jointly updates observed states by EnSF before regression; this differs from serial per-observation update schemes in some particle flow filters.
• EnSF-LR's hybrid nonlinear and linear regression update provides a natural framework for partially observed nonlinear DA.

Threat model

n/a — this paper addresses methodological and algorithmic advances in data assimilation filtering rather than adversarial security scenarios.

Methodology — deep read

1. Threat Model & Assumptions: The problem is sequential Bayesian filtering in partially observed nonlinear chaotic systems where only a subset of the state vector can be directly observed via a potentially nonlinear observation operator. The adversary consideration is n/a. This is a methodological DA research study focusing on accurate filtering with sparse, nonlinear observations. The model assumes additive Gaussian observation noise and perfect model dynamics without stochastic model error.

2. Data: Two standard chaotic benchmark systems are used - the 3D Lorenz-63 and the 40D Lorenz-96 models. Sparse observation networks are used: in Lorenz-63 only the x component is observed; in Lorenz-96 every fourth state variable is observed (25% coverage). Both linear (identity) and nonlinear (arctan) observation operators are tested. Observations are assimilated every 10 model integration steps. The truth trajectories are generated by integrating the models and using a spin-up phase; ensembles are sampled from attractor states to ensure realistic spread.

3. Architecture / Algorithm: EnSF is a training-free, diffusion-based ensemble score filter that approximates prior score from the forecast ensemble and uses a pseudo-time score-based update to incorporate observations. The novelty is the two-step EnSF-LR: (i) apply EnSF update only to observed state components using the score-based nonlinear update (Eq. 10), leaving unobserved components unchanged; (ii) compute observed-state analysis increments and propagate these to unobserved components via linear regression using ensemble-estimated covariances Puo and Poo (Eq. 13), analogous to the regression step in EnKF. This hybrid approach leverages nonlinear generative modeling for observed variables and linear covariance-informed regression for unobserved variables.

4. Training Regime: No training involved; EnSF approximates prior score directly from forecast ensemble. Large ensemble sizes of Ne=1000 for Lorenz-63 and Ne=10,000 for Lorenz-96 are used to reduce sampling errors. The EnSF pseudo-time integration uses 500 uniform steps. Assimilation is performed over ~300-600 steps.

5. Evaluation Protocol: Performance is measured by root-mean-square error (RMSE) of the analysis ensemble mean versus the true state, computed over multiple assimilations and averaged over 20 independent realizations with 95% confidence intervals reported. Comparisons are made against the original EnSF and a stochastic EnKF baseline, which is equivalent mathematically to the two-step regression formulation. Both linear and nonlinear observation cases are evaluated for Lorenz-63 and Lorenz-96.

6. Reproducibility: The paper does not explicitly announce code or data release. The benchmark models and observation setups are publicly known standard prototypes; however, ensemble sizes and certain algorithm parameters (e.g. pseudo-time discretization) are detailed for replication. The perfect model assumption and no stochastic model error simplify reproducibility.

Example end-to-end (Lorenz-63 nonlinear observation): At each assimilation step, the forecast ensemble (Ne=1000) is propagated. Observations of x processed through nonlinear arctan operator with Gaussian noise. EnSF update performs score-based nonlinear correction on observed x only. Observed-state increments computed. Prior cross-covariance used to linearly regress increments onto unobserved y and z components, updating those states. The combined updated ensemble forms the posterior, which is then advanced to next cycle. Over multiple cycles, RMSE improves significantly relative to EnSF and EnKF alone.

Technical innovations

• Introduction of a two-step hybrid filtering approach (EnSF-LR) combining nonlinear score-based updates on observed variables with covariance-based linear regression for unobserved variables.
• Application of ensemble-based prior cross-covariance to map observed-state nonlinear analysis increments onto unobserved components, improving corrections under sparse observations.
• Use of training-free EnSF score estimation within a partially observed setting by isolating likelihood-gradient corrections to observed subspace.
• Demonstration that EnSF-LR bridges the performance gap between purely nonlinear score-based filtering and classical EnKF under both linear and nonlinear observation operators.

Datasets

• Lorenz-63 — 3-dimensional chaotic model — synthetic benchmark
• Lorenz-96 — 40-dimensional atmospheric toy model — synthetic benchmark

Baselines vs proposed

• Original EnSF: full-state RMSE ≈ 6.3 (Lorenz-63 nonlinear obs) vs EnSF-LR: ≈ 2.7
• EnKF (stochastic): full-state RMSE ≈ 4.2 (Lorenz-63 nonlinear obs) vs EnSF-LR: ≈ 2.7
• Original EnSF: full-state RMSE higher than EnKF and EnSF-LR in linear observations (Fig. 1)
• In linear sparse observation experiments, EnSF-LR reduces EnSF RMSE substantially and approaches EnKF accuracy (Fig. 1)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.28264.

Fig 1: Analysis-time full-state RMSE for the linear-observation experiments in (a) Lorenz-63 and (b) Lorenz-

Fig 2: Observed- and unobserved-state RMSE for the Lorenz-63 nonlinear-observation experiment: (a) observed

Fig 3: Representative ensemble trajectories for the Lorenz-63 nonlinear-observation experiment. Rows show the

Fig 4: Three-dimensional reference trajectory of the Lorenz-63 system for the representative nonlinear-

Fig 5: Forecast and analysis ensemble scatter plots in the X–Y plane for the Lorenz-63 nonlinear observation

Fig 6: Analysis-time full-state RMSE for the nonlinear-observation experiments in (a) Lorenz-63 and (b) Lorenz-

Fig 7: Observed- and unobserved-state RMSE for the Lorenz-96 nonlinear-observation experiment: (a) observed

Fig 8: Representative ensemble trajectories for the Lorenz-96 nonlinear-observation experiment. Rows show the

Limitations

• Experiments use very large ensembles (1000 to 10,000 members), so performance with practically sized ensembles is unknown.
• No model error or stochastic model uncertainty considered; perfect model assumption simplifies dynamics and reduces realism.
• No systematic exploration of observation network design or localization and inflation effects for small ensembles.
• The regression step is performed jointly, not in a serial assimilation framework, leaving implementation in real large-scale operational systems open.
• No adversarial or robustness testing under model misspecification, biased observations, or adversarial noise.

Open questions / follow-ons

• How well does EnSF-LR perform with smaller ensemble sizes common in operational DA settings when localization and inflation are required?
• Can the two-step hybrid framework be extended and efficiently implemented for high-dimensional geophysical models with complex, heterogeneous observation networks?
• How robust is EnSF-LR to model error, observation bias, or missing data scenarios often present in real-world assimilation problems?
• Can the regression update step be integrated into a serial or asynchronous observation assimilation framework, allowing scalable operational implementation?

Why it matters for bot defense

While not directly related to CAPTCHA or bot-defense, the core idea of propagating partial observational corrections across related but unobserved components via learned or estimated statistical relationships could inspire more robust behavioral or feature-based anomaly detection approaches in security contexts. In particular, EnSF-LR’s hybrid nonlinear plus linear regression framework offers an example of leveraging generative modeling for observed data combined with covariance-informed inference of unobserved latent variables. For CAPTCHA/anti-bot engineers, the lesson is that hybrid inference schemes that incorporate nonlinear machine-learned updates with classical ensemble statistics may enhance estimation accuracy when direct measurements (e.g., user behavior signals) are sparse or incomplete. Also, the two-step filtering perspective clarifies how to manage partial observations and their latent extensions in real-time prediction or classification pipelines.

Cite

@article{arxiv2606_28264,
  title={ A Two-Step Ensemble Score Filter for Data Assimilation in Partially Observed Systems },
  author={ Zixiang Xiong and Feng Bao and Hristo G. Chipilski and Siming Liang and Jingqiao Tang and Guannan Zhang },
  journal={arXiv preprint arXiv:2606.28264},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.28264}
}

Read the full paper

• arXiv:2606.28264 (abstract)
• arXiv:2606.28264 (PDF)