NestyNet. II. Coherent Function-Space Posteriors from Scientific Neural Surrogates (or How to Avoid Expensive MCMC)
Source: arXiv:2608.06274 · Published 2026-08-06 · By Rodrigo Ibata, Wassim Tenachi, Foivos Diakogiannis, Neil Ibata, Anirudh Shankar
TL;DR
This paper addresses the challenge of quantifying uncertainty in scientific neural surrogate models that often have thousands of correlated parameters, which makes traditional parameter-space uncertainty estimation impractical and uninterpretable. The authors propose constructing a low-dimensional function-space posterior by linearizing the fitting procedure around a converged solution using the Gauss–Newton curvature and Jacobians, resulting in a measurement-to-function transport operator. Decomposing this operator via singular value decomposition yields coherent deformation modes that span the uncertainty space efficiently. Independent Gaussian draws in this reduced basis generate smooth function samples that can propagate uncertainty through any differentiable derived scientific quantities. This approach avoids expensive Markov chain Monte Carlo (MCMC) sampling of high-dimensional parameter spaces.
The method is conditioned on fixed model architecture, hyperparameters, and the converged optimization branch, and includes diagnostics to certify convergence and detect breakdowns of the linear approximation. In a key example, the authors fit an 800-parameter stellar phase-space distribution function and demonstrate that just 4 uncertainty coordinates capture 99% of the posterior variance in vertical-force predictions, enabling rapid propagation of 4000 coherent function draws through downstream dynamical calculations in under a second. Validation on a smaller, analytically tractable rotation curve example shows excellent agreement with exact spline posteriors and MCMC sampling. Overall, this work provides a compact, interpretable, computationally efficient uncertainty quantification framework for large scientific neural surrogates enabling uncertainty propagation for derivative-dependent inference without costly high-dimensional sampling.
Key findings
- Linearizing the fitting procedure yields a measurement-to-function transport operator whose leading singular functions define low-dimensional coherent function-space posterior modes.
- In an 800-parameter stellar phase-space fit, 4 uncertainty modes capture 99% of the vertical-force posterior variance.
- 4000 coherent draws propagated through downstream calculations (force, total density, surface density, frequency) complete in 0.8 seconds.
- Validation on a 18-parameter mock rotation curve shows NestyNet's function-space posterior matches exact spline Gaussian posteriors and ensemble MCMC histograms to within a few percent in width and center.
- Posterior correlation matrices in function or deliverable space reveal clear physical structure, unlike dense, gauge-dependent parameter correlation matrices.
- A tangent-adequacy diagnostic Q detects nonlinear response directions where the linearized Laplace approximation breaks down, preventing silent failure.
- The approach distinguishes repeated-experiment sampling covariance from local Gauss-Newton/Laplace posterior uncertainty.
- Conditioning on fixed hyperparameters, architecture, and optimized branch is critical to defining a coherent posterior representation.
Threat model
Not a security threat paper; adversarial context is absent. The 'adversary' is measurement noise and uncertainty propagation in scientific neural surrogate models. The adversary cannot perform arbitrary nonlinear refits or change the optimizer state but can perturb measured residuals within known covariance. Conditioning on fixed optimizer branch and hyperparameters means uncertainty quantification excludes model choice and optimization ambiguity.
Methodology — deep read
The paper first defines the threat model as scientific inference using neural surrogates with deterministic second-order optimizers that yield converged stationary points conditional on chosen architecture, hyperparameters, and optimization branch. The adversary is not an attacker but uncertainty from measurement noise and modeling choices; uncertainties stem from randomized residuals in data.
Data used include synthetic mock datasets such as an 18-parameter rotation curve model with 50 noisy radius-velocity points, and a larger 800-parameter stellar phase-space distribution function fit. Labels are function values; splits or preprocessing are deterministic network fits and whitening via measurement covariance.
The core algorithm is to linearize the stationary optimum conditions w.r.t. measurement perturbations, yielding the measurement-to-function transport operator TX = J_f,H^{-1} J_m^T which maps whitened residual perturbations to function perturbations. Here J_m and J_f are Jacobians at the converged solution and H is the Gauss-Newton curvature matrix (Jacobian transpose times Jacobian plus persistent prior residual curvature). Truncation of small eigenvalues of H reduces rank.
An SVD of TX weighted by output metric (e.g. quadrature weights) produces singular functions psi_k and singular values s_k, giving a Karhunen-Loève-optimal low-dimensional decomposition of function-space posterior modes. A generative model is formed as F_zeta(x) = f_hat(x) + sum_k s_k zeta_k psi_k(x) with zeta_k independent standard normals. This yields coherent smooth random draws of functions respecting correlations and differentiability.
Training is replaced by deterministic second-order optimization to convergence on fixed data and hyperparameters, with no stochastic gradient descent or sampling involved. Fits are certified converged using normed Gauss-Newton residual corrections below thresholds. No sampling of parameter posterior is done—only linearized curvature solves and Jacobian-vector products.
Evaluation includes quantitative validation by comparing function-space posteriors, credible intervals, and derived quantity distributions against independent spline model closed-form posteriors and affine-invariant ensemble MCMC on the rotation curve example. Posterior covariance matrices in function/deliverable space are analyzed, and tangent-adequacy ratio Q diagnostics test linear approximation validity. Ablations examine conditioning on fixed hyperparameters, active sets, and single optimization branches.
Reproducibility: Jacobians, residuals and curvature matrices come from the previously described NestyNet second-order segmented optimization from Paper I. Code and weights are not detailed herein. The datasets are synthetic mock data with described parameters. All linear algebra steps are explicit and deterministic given the fit.
An end-to-end example is the rotation curve case where 18 parameters define a segmented neural fit to 50 noisy radial velocities; the measurement-to-function transport operator is constructed by evaluating converged Jacobians, curvature, and prediction Jacobians. The SVD modes psi_k are computed on the output radius grid with quadrature weights. Function draws are synthesized using these modes and Gaussian coefficients, producing posterior credible bands which closely match exact spline Gaussian posteriors and MCMC histograms. Derived physical quantities like dynamical mass and epicyclic frequency are computed with analytic gradients, inheriting the posterior uncertainty. The tangent-adequacy ratios Q are evaluated to identify directions where the linearization fails, prompting further checks or refits.
Overall, the method replaces expensive repeated MCMC or resampling for high-dimensional neural surrogates with a deterministic linearized Laplace-like posterior in function space, represented sparsely by coherent deformation modes derived from converged optimizer Jacobians and curvatures. This posterior enables fast coherent draws and credible intervals on predictive and derived quantities including derivatives and integrals.
Technical innovations
- Construction of a measurement-to-function transport operator from converged Jacobians and Gauss–Newton curvature to linearize refitting in function space.
- Singular value decomposition of this transport yields a Karhunen–Loève-optimal low-dimensional coherent basis of posterior deformation modes.
- Separation of repeated-experiment covariance from the local Laplace/Gauss–Newton posterior, enabling distinct uncertainty quantification.
- A tangent-adequacy diagnostic Q to detect breakdown of linearized approximation along posterior directions preventing silent failures.
- End-to-end uncertainty propagation to derived scientific quantities involving derivatives and integrals using the same functional draws.
Datasets
- Mock rotation curve — 50 radial velocity points — synthetic exponential disk with Gaussian noise
- Mock stellar disk phase-space distribution — 800-parameter fit synthetic data — described in text
Baselines vs proposed
- Spline + MCMC: posterior interval width on mass proxy M = 0.0286–0.0296 vs NestyNet sister draws: 0.0286–0.0296
- Exact spline closed-form Gaussian posterior: center at 0.8612 vs NestyNet sister mean 0.8631 with matching credible bands
- Posterior correlation matrix in parameter space: dense and uninterpretable (median |ρ|=0.65) vs deliverable space: physically meaningful correlations (e.g., ρ(κ, B)=-0.99)
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.06274.
Fig 1 (page 1).
Fig 1: Controlled validation of the sister on a mock rotation curve, and one local posterior represented in two languages.
Fig 3 (page 7).
Fig 2: The transport tested against fully converged nonlinear refits (§5.6): 48 parametric mock datasets refit warm from
Fig 3: The vertical-force and local-density tower from the evidence-selected α = 0.125, P = 800 exact-Poisson fit, and the
Limitations
- Method requires deterministic converged optimizer fit and conditions on architecture, hyperparameters, optimization branch—uncertainty from these choices is not marginalized.
- Linearization assumes the Gaussian approximation near the optimum; nonlinear/posterior curvature beyond Gauss–Newton could cause inaccuracies.
- Diagnostic Q flags some directions of nonlinear response, but explicit refits or higher-order corrections are needed there; no full nonlinear posterior sampling is done.
- No explicit test on real data or adversarial examples; demonstrated on synthetic mocks only.
- No release of code or frozen weights, limiting reproducibility claims outside the described environment.
- Approach focuses on uncertainties pertaining to measurement noise and fixed prior curvature, not model misspecification or distributional shifts.
Open questions / follow-ons
- How to incorporate uncertainty from model architecture, hyperparameter selection, or optimization branch changes into the function-space posterior framework?
- Can the linearized Laplace approximation and transport be extended to better capture or correct for nonlinear posterior features beyond local Gaussianity?
- How well does this method scale and perform on real scientific datasets with complex noise structures and multiple sources of uncertainty?
- Could the tangent-adequacy diagnostic be generalized or automated to trigger adaptive higher-order corrections or selective sampling?
Why it matters for bot defense
Bot-defense and CAPTCHA practitioners can draw insights from this paper's function-space uncertainty quantification approach to better understand and model uncertainties in complex, high-dimensional neural surrogates—analogous to CAPTCHA solvers or behavior models. The methodology’s emphasis on coherent low-dimensional uncertainty representations and linearized transport operators may inspire scalable uncertainty quantification or anomaly detection methods in CAPTCHA systems without costly high-dimensional Monte Carlo sampling. Additionally, the diagnostic tools to detect breakdowns of linear approximations could help flag situations where model assumptions fail, which is critical in real-time security settings.
However, the paper’s scientific inference context with known, fixed architecture and converged second-order optimization differs from dynamic adversarial environments in bot defense. CAPTCHA engineers should be cautious applying these techniques directly, since conditioning on fixed model and hyperparameters may overlook uncertainty arising from adaptive adversaries or deliberate manipulation. Still, the principled decomposition of uncertainty and efficient propagation of function-space posteriors offers a compelling framework for interpretable, computationally efficient uncertainty management in complex deployed neural systems.
Cite
@article{arxiv2608_06274,
title={ NestyNet. II. Coherent Function-Space Posteriors from Scientific Neural Surrogates (or How to Avoid Expensive MCMC) },
author={ Rodrigo Ibata and Wassim Tenachi and Foivos Diakogiannis and Neil Ibata and Anirudh Shankar },
journal={arXiv preprint arXiv:2608.06274},
year={ 2026 },
url={https://arxiv.org/abs/2608.06274}
}