Hybrid SINDy-EnKF in Learning Chikungunya Dynamics from Incomplete, Noisy or Partially Observed Data

Source: arXiv:2607.27137 · Published 2026-07-29 · By Bernard Asamoah Afful, Changhong Mou, Luis Gordillo

TL;DR

This paper addresses the challenge of learning reliable mechanistic models for Chikungunya virus (CHIKV) transmission dynamics from noisy, incomplete, and partially observed epidemiological data. Traditional compartmental epidemic models require accurate parameterization and fully observed states, which are rarely available in practice, limiting prediction and interpretability. The authors propose a novel hybrid data-driven framework that couples Sparse Identification of Nonlinear Dynamics (SINDy)—an algorithm that discovers sparse governing equations from time series data—with the Ensemble Kalman Filter (EnKF), a sequential data assimilation method that filters noise and estimates unobserved states online.

The method first uses SINDy to discover the underlying differential equations from available noisy data and then applies EnKF to sequentially assimilate new observations, correcting SINDy predictions and reconstructing unobserved compartments. Experiments on a detailed 10-compartment CHIKV host-vector model show that standalone SINDy accurately recovers the model from noise-free data but is highly sensitive to observational noise, with performance degrading rapidly above 10% noise. Integrating EnKF substantially improves state estimation accuracy, especially under partial observability typical of surveillance, reconstructing unobserved compartments with normalized RMSE below 0.1 and correlation exceeding 0.95 under realistic noise conditions. This hybrid approach balances interpretability, data-driven flexibility, and noise robustness, offering a promising direction for learning epidemiological dynamics from challenging data.

Key findings

• SINDy accurately reconstructed the full 10-compartment CHIKV system with component-wise RMSE below 1 and relative RMSE ~10^-5 in noise-free simulations (Section 3.1, Tables 4 and 5).
• SINDy performance quickly degrades as additive Gaussian noise rises, with component-wise RMSE increasing over three orders of magnitude at 25% noise and relative RMSE reaching ~0.17 at 50% noise (Sections 3.2.2-3.2.3, Table 5).
• Noise amplification in numerical differentiation caused SINDy to retain spurious terms or lose genuine couplings under noise, leading to biologically implausible oscillations or unbounded growth in some compartments (Section 3.2.3).
• Applying EnKF filtering to the SINDy-discovered model enabled accurate online state estimation from noisy, partial observations, reconstructing unobserved compartments with normalized RMSE (NRMSE) < 0.1 and temporal correlation > 0.95 when observation intervals were ≤ 1 day (Section 3.3.2, Figure 7).
• Lower frequency of observations (dtobs > 1 day) caused degradation in EnKF accuracy for unobserved states, showing the importance of observation density for data assimilation performance (Section 3.3.2).
• Even with only host data observed (vaccinated Vh, infectious Ih, treated Jh, recovered Th), EnKF effectively recovered vector states (Sv, Ev, Iv) and unobserved host states (Sh, Eh, Rh), demonstrating the capability to infer hidden variables (Section 3.3.1).
• The hybrid approach successfully used a mass-action bilinear candidate library in SINDy, leveraging epidemiological domain structure for interpretability and sparse recovery (Section 3.1).
• Single global sparsity threshold in SINDy was insufficient under noise due to scale disparities across compartments, motivating compartment-wise or weighted thresholding for improved robustness (Section 3.2.3).

Threat model

n/a — This paper focuses on epidemiological model identification and state estimation under noisy and partial observations, with no explicit adversary or security threat assumptions.

Methodology — deep read

1. Threat Model & Assumptions: The adversary context is not applicable as this is a modeling study. The focus is on realistic epidemiological surveillance scenarios where data are incomplete, noisy, or observed only partially. Noise is modeled as additive zero-mean Gaussian with known covariance. The challenge is to recover latent dynamics when only some compartments are observed and observations contain noise.

2. Data: Synthetic data are generated by numerically integrating a 10-compartment ODE CHIKV host-vector model with parameters chosen from epidemiological literature. Initial conditions and parameters are fixed as in Tables 2 and 3. The state vector x spans 10 compartments including vaccinated, infected (symptomatic, asymptomatic, treated), susceptible, exposed, recovered hosts, and mosquito states. Time series of length m with fine time step are generated. Noise is added at various relative amplitudes (5%-50%) to simulate observational error. Partial observability is simulated by restricting observed compartments to a subset (Vh, Ih, Jh, Th).

3. Architecture / Algorithm: The Sparse Identification of Nonlinear Dynamics (SINDy) framework is applied to data x(t), constructing a library Θ(x) of candidate nonlinear functions including constants, linear and bilinear terms reflecting mass-action infection dynamics. SINDy performs sequentially thresholded least squares (STLSQ) regression to find a sparse coefficient matrix Ξ producing a sparse set of governing ODEs dx/dt = Θ(x)Ξ. In noisy conditions, SINDy applied directly to numerical derivatives of observed noisy data suffers from noise amplification and spurious terms. To mitigate this, they embed the SINDy-discovered model as a forecast operator within an Ensemble Kalman Filter (EnKF) framework. The EnKF propagates an ensemble of state estimates forward via the SINDy ODE forecast, then assimilates noisy partial observations to update the state using Kalman gain computed from ensemble covariances. This corrects noise-corrupted forecasts, infers unobserved compartments, and improves robustness in realistic scenarios.

4. Training Regime: Not a training-based model in the ML sense. SINDy regressions use STLSQ with sparsity thresholds tuned empirically by noise level. Ensemble Kalman filtering uses an ensemble of Ne members initialized with state perturbations around true initial condition. Time discretization uses a fixed time step dt, with assimilation cycles varying in frequency (dtobs from 0.1 to 10 days). No stochastic learning or iterative gradient updates; pipeline is model-discovery followed by sequential filtering.

5. Evaluation Protocol: Performance metrics include component-wise root mean square error (RMSE) relative to ground truth trajectories, relative RMSE (Frobenius norm), normalized RMSE (NRMSE), and temporal correlation between estimates and true states. Ablations include varying noise amplitude (5%-50%) and partial observability levels. Sensitivity to observation frequency was assessed by changing filtering assimilation interval dtobs. Baselines include pure SINDy reconstruction from noise-free and noisy data without filtering. Statistical tests were not reported. All experiments use synthetic data generated from known model.

6. Reproducibility: Code or data release not mentioned; the CHIKV model is standard and parameters are fully specified. SINDy and EnKF algorithms used are standard in literature, enabling straightforward reproduction. Details on hyperparameters (e.g., sparsity thresholds for each noise level) are reported, enhancing replicability.

Example: In one experiment, clean CHIKV trajectories were used to build the derivative matrix; SINDy recovered equations with near-zero error. When 10% Gaussian noise was added to observed states and only four host compartments were observed, the SINDy model was used as forecast operator inside EnKF. The EnKF assimilated noisy data every 1 day, maintaining NRMSE below 0.1 and correlation above 0.95 for both observed and inferred states over the integration horizon, effectively reconstructing the full system despite partial observations and noise.

Technical innovations

• Coupling SINDy-derived epidemic models with an ensemble Kalman filter to jointly correct forecasts and infer unobserved epidemiological states from noisy, partial observations.
• Systematic noise characterization revealing the limitations of pure SINDy for epidemiological data with realistic noise levels and motivating data-assimilation enhancement.
• Application of mass-action bilinear candidate function libraries within SINDy tailored to host-vector compartment models for interpretable sparse model discovery.
• Demonstration that EnKF can robustly reconstruct unobserved compartments in complex infectious disease dynamics by exploiting cross-covariances from partial observations.

Datasets

• Synthetic CHIKV host-vector compartment trajectories — size unspecified but derived from ODE integration over 350+ days — generated in-house, not publicly released

Baselines vs proposed

• SINDy (noise-free): component-wise RMSE < 1 vs noisy data RMSE up to several thousand (Table 4)
• Relative RMSE (rRMSE) SINDy noise-free: ~1e-5 vs 0.17 at 50% noise (Table 5)
• SINDy with EnKF data assimilation (dtobs ≤ 1): NRMSE < 0.1 and correlation > 0.95 for observed and unobserved states (Section 3.3.2)
• SINDy alone (≥25% noise): rapid performance degradation with biologically implausible dynamics vs SINDy-EnKF stable reconstructions

Limitations

• All data and evaluation are performed on synthetic simulations from a single predefined CHIKV model with fixed parameters; real-world epidemiological data might differ significantly.
• The noise model assumes Gaussian, state-dependent additive noise with known covariance, which may not capture complex errors in surveillance data.
• Performance degrades substantially when observations are infrequent (dtobs > 1 day), a common scenario in practice that may limit applicability.
• The study uses a single global sparsity threshold in SINDy; more sophisticated compartment-wise or adaptive regularization was not implemented but suggested as future work.
• No adversarial, distribution shift, or model misspecification robustness analyses were reported.
• Code and trained models are not publicly released, limiting immediate reproducibility.

Open questions / follow-ons

• Can compartment-specific or adaptive sparsity regularization in SINDy further improve noise robustness and interpretability in epidemic modeling?
• How well does the hybrid SINDy-EnKF framework transfer to real-world epidemiological data with complex, non-Gaussian noise and reporting delays?
• What are the limits of partial observability and observational frequency under which accurate state and parameter reconstruction remain feasible?
• Can the method be extended to handle structural model mismatch or unknown external forcing dynamically during assimilation?

Why it matters for bot defense

For bot-defense or CAPTCHA practitioners, this paper is primarily relevant as a case study in combining data-driven sparse model discovery with sequential filtering under noisy, partial observability constraints. Although this focuses on epidemiological dynamics, the principles of correcting learned nonlinear dynamics with ensemble filtering apply broadly to bot detection challenges where system states may be partially observed and observations noisy.

Specifically, the hybrid SINDy-EnKF approach demonstrates how to recover interpretable dynamic models that propagate unknown internal states, then correct those predictions online via noisy measurement assimilation. This paradigm could inform designs for robust bot behavior modeling, tracking latent botnet states, or improving detection accuracy from partial behavioral signals in noisy real-world environments. The sensitivity to observation frequency and noise level also provides caution that learning-based defenses require adequate data density and noise mitigation strategies, motivating data-assimilation approaches rather than standalone regression. Lastly, the emphasis on interpretable sparse models aligns with bot-defense needs for explainability and auditability.

Cite

@article{arxiv2607_27137,
  title={ Hybrid SINDy-EnKF in Learning Chikungunya Dynamics from Incomplete, Noisy or Partially Observed Data },
  author={ Bernard Asamoah Afful and Changhong Mou and Luis Gordillo },
  journal={arXiv preprint arXiv:2607.27137},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.27137}
}

Read the full paper

• arXiv:2607.27137 (abstract)
• arXiv:2607.27137 (PDF)