Multimodal Empirical Bayes Variational Autoencoders for Joint Longitudinal and Time-to-Event Modeling

Source: arXiv:2607.13984 · Published 2026-07-15 · By Anders Sjöberg, Nils Olsson, Marcus Baaz, Mats Jirstrand

TL;DR

This paper addresses the challenge of jointly modeling longitudinal tumor growth data and time-to-event outcomes such as dropout within a unified probabilistic framework. It extends the empirical Bayes variational autoencoder (EB-VAE) approach to incorporate a hazard model alongside longitudinal trajectories and introduces conditioning of the prior on genomic covariates. This allows the model to capture individual variability modulated by treatment and genetic factors. Two decoder architectures are developed: a fully neural NODE-based decoder and a hybrid semi-mechanistic decoder using exponential tumor growth with a NODE hazard function. Evaluated on large patient-derived xenograft (PDX) datasets from cutaneous melanoma and breast cancer, the framework accurately reproduces tumor volume distributions and dropout patterns in held-out individuals. The hybrid decoder yields tumor growth parameters close to literature nonlinear mixed-effects model estimates, while the neural decoder delivers improved predictive accuracy. Incorporating genetic covariates reduces prediction error and stability selection highlights biologically plausible genetic indicators. Overall, the work demonstrates a flexible, interpretable, and scalable latent variable approach for integrating multimodal pharmacometric data within joint longitudinal and time-to-event modeling.

Key findings

• The Hybrid-EB-VAE recovered baseline tumor growth rate kg = 0.0830 vs reference 0.06, and monotherapy treatment effects (e.g. LEE011 = 0.0372 vs reference 0.0444) consistent with classical NLME models.
• Both Neural-EB-VAE and Hybrid-EB-VAE achieved similar prior RMSE on held-out individuals for tumor volume prediction (195 vs 196). Neural-EB-VAE had lower posterior RMSE (68 vs 97).
• Neural-EB-VAE joint model accurately captured dropout and tumor volume distributions, outperforming a naive threshold dropout model with biased predictions (Fig 3).
• Genetic prior conditioning reduced individual RMSE by approximately 13% on 21-day cutaneous melanoma data (from 161 to 140) with 63% of individuals showing improvement.
• Stability selection identified biologically relevant genetic indicators such as BRAF V600E, NRAS, NF1, and MDM2 mutations associated with tumor growth and treatment response.
• Joint training across multiple treatment groups improved prediction accuracy compared to training on single treatment groups alone, especially for untreated, dacarbazine, and encorafenib (Fig 6).
• The hazard-based dropout component modeled time-to-event jointly with longitudinal tumor growth, coupling through shared latent variables.
• The hybrid decoder offered interpretable tumor growth dynamics parameters while the neural decoder provided flexible nonlinear dynamics suitable for treatment generalization.

Threat model

n/a — This pharmacometric modeling paper does not address adversarial threat models or security-related attacks. The focus is on statistical modeling of heterogeneous biomedical data under observational uncertainty rather than threat scenarios.

Methodology — deep read

1. Threat Model & Assumptions: The model assumes an adversary indirectly in the pharmacometric setting—typically no explicit adversary but the framework must robustly model inter-individual variability and informative dropout phenomena across patients without overfitting. The adversary is effectively uncertainty due to population heterogeneity, missingness patterns, and high-dimensional genomic data. No explicit security threats are considered.

2. Data: The main data source is a large-scale patient-derived xenograft (PDX) dataset from multiple solid tumor types, focusing on cutaneous melanoma (CM) and breast cancer (BC). The 60-day 6-treatment CM dataset includes 183 individuals across 6 treatments, with about 11 longitudinal tumor volume observations per individual. Larger 21-day and 100-day CM/BC datasets range up to 826 individuals and 22 treatment groups, incorporating genomic data including gene mutations, copy-number alterations, and RNA expression. Individuals with missing genetic data were excluded in genetic experiments. Observations were truncated at predefined cutoffs (21, 60, or 100 days).

3. Architecture & Algorithms: EB-VAE framework with encoder-decoder architecture models inter-individual variability (IIV) via latent variables ki regularized by a covariate-conditioned empirical Bayes prior pψ(ki | xi). The encoder network qϕ(ki | yi, xi) performs amortized variational inference, mapping longitudinal tumor measurements yi and covariates xi to a latent space. The decoder maps latent representation and treatments to tumor volume trajectories. Two decoder variants:

• Neural-EB-VAE: all decoder functions gθ, fθ, hθ implemented as neural networks and Neural ODEs for flexible nonlinear dynamical modeling.
• Hybrid-EB-VAE: mechanistic exponential tumor growth model dzi/dt = kizi(t) for tumor dynamics, with ki as individual effective growth rates modulated by treatment; hazard function for dropout modeled by NODE.

The joint model treats longitudinal observations and time-to-event dropout as conditionally independent given latent effects, coupling them through a shared latent state and hazard integral. The hazard λθ depends on hidden states and treatments. The cumulative hazard Hi(t) is integrated as an augmented ODE state alongside latent tumor state zi(t).

Genomic covariates are encoded as high-dimensional binary vectors representing mutations and copy number changes curated from literature, databases, and AI-assisted candidate generation. Due to sparsity and overfitting risk, genetic conditioning is done via a two-stage approach: first training EB-VAE without genetics, then fixing encoder, decoder, and treatment prior, training a neural shift model that adapts the prior distribution in latent space based on genetic covariates.

4. Training Regime: Training optimizes a joint evidence lower bound (ELBO) combining longitudinal reconstruction likelihood, variational KL regularization with empirical Bayes prior, and time-to-event log-likelihood. Dropout censoring is incorporated into likelihood. Adam optimizer is applied; specific epoch or batch size details were not disclosed. Genetic conditioning involves separate training stages with fixed decoder.

5. Evaluation Protocol: Performance metrics include individual-level RMSE of tumor volume prediction (median prior/posterior predictive trajectories vs observed), population-level predictive checks via prediction-corrected visual predictive checks (pcVPC) for tumor volumes, Kaplan–Meier visual predictive checks (KM–VPC) for dropout survival functions, and treatment stratified VPCs. Five-fold cross-validation over held-out individuals and treatment groups tests generalization. Stability selection ranks genetic feature importance by perturbing individual indicators and measuring effects on validation KL divergence.

6. Reproducibility: Code and detailed hyperparameters were not explicitly mentioned as released. Dataset is proprietary PDX data but references publicly documented sources and algorithms. The EB-VAE framework was previously introduced by the authors in [18].

Example: For an individual PDX tumor in the 60-day CM dataset, the encoder uses observed longitudinal tumor volumes up to day 60 and covariates (treatment group, possibly genetics) to infer a posterior distribution qϕ(ki). A latent vector ki is sampled and fed to the decoder, which simulates tumor growth trajectory zi(t) via NODE or mechanistic ODE along with cumulative hazard Hi(t) for dropout. The model evaluates tumor volumes ˆyi(t) and event likelihoods jointly, optimized via ELBO. Predictions and dropout estimates are compared with held-out longitudinal and event times for that individual.

Technical innovations

• Extension of EB-VAE framework to jointly model longitudinal tumor growth and time-to-event dropout via integration of NODE-based hazard modeling within decoder dynamics.
• Introduction of hybrid mechanistic–neural decoder combining classical exponential tumor growth ODEs with NODE-based hazard function for interpretable pharmacometric parameters.
• Two-stage genetic conditioning where a genetics-conditioned empirical Bayes prior is learned via neural latent space shifts without retraining the entire encoder-decoder dynamics.
• Application of stability selection on genetics-conditioned latent prior to identify biologically plausible genetic markers influencing tumor progression and dropout.

Datasets

• 60-day 6-treatment cutaneous melanoma (CM) — 183 individuals, 6 treatments, ~11 observations per individual — proprietary PDX data
• 21-day cutaneous melanoma (CM) — 516 individuals, 17 treatments, ~6 observations per individual — proprietary PDX data
• 100-day cutaneous melanoma (CM) — 516 individuals, 17 treatments, ~11.6 observations per individual — proprietary PDX data
• 21-day breast cancer (BC) — 826 individuals, 22 treatments, ~6.4 observations per individual — proprietary PDX data
• 100-day breast cancer (BC) — 826 individuals, 22 treatments, ~14.4 observations per individual — proprietary PDX data

Baselines vs proposed

• Hybrid-EB-VAE vs Neural-EB-VAE: prior RMSE = 195 vs 196; posterior RMSE = 97 vs 68
• Neural-EB-VAE vs naive threshold dropout model: KM-VPC and pcVPC show naive dropout model bias in predicting dropout and tumor volume distributions, Neural-EB-VAE more accurate (Fig 3).
• Model without genetics vs with genetics (21-day CM): mean RMSE = 161 vs 140 (13% reduction) with genetic conditioning.
• Single-treatment group model vs multi-treatment joint model (21-day CM): multi-treatment training reduces RMSE across most treatment groups, e.g., untreated group improves significantly (Fig 6).

Limitations

• The tumor growth mechanistic model is limited to exponential growth, may not capture more complex tumor dynamics in all cases.
• Genetic conditioning uses a two-stage approach without end-to-end training, possibly limiting integration fidelity between genetic effects and longitudinal dynamics.
• Datasets are derived from PDX models and proprietary; generalizability to clinical patient populations is unclear.
• No explicit adversarial or out-of-distribution robustness evaluations were performed.
• Stability selection provides hypothesis-generating genetic indicators but does not establish causal effects or predictive utility in independent cohorts.
• The training hyperparameters and exact computational resources are not fully disclosed, impacting reproducibility.

Open questions / follow-ons

• How well would the EB-VAE framework generalize to clinical trial or real-world clinical patient data with noisier measurements and more heterogeneous dropout mechanisms?
• Could an end-to-end joint training strategy incorporating genetic conditioning alongside decoder dynamics further improve prediction and mechanistic interpretability?
• How robust are the genetic indicator findings to alternative genomic preprocessing, additional omics modalities, or causal inference frameworks?
• Would more complex mechanistic models beyond exponential growth improve predictive accuracy and clinical interpretability?

Why it matters for bot defense

While this work primarily addresses biomedical pharmacometric modeling, its core methodological advances in joint latent-variable modeling of multimodal temporal and event data could inspire bot-defense researchers. For CAPTCHA and bot-defense systems, integrating heterogeneous time-series signals (user interactions, event sequences) with structured priors conditioned on user metadata or environment covariates shares conceptual similarity. EB-VAE's modular framework enables interpretable latent representations capturing subtle dependencies between longitudinal behaviors and dropouts (failures), which aligns with detecting early signals of bot activity or session termination. The use of hazard-based dropout modeling might analogously represent bot session abandonment or challenge failure. However, direct application requires adapting the domain-specific decoder components and covariate sets. The two-stage conditioning approach for high-dimensional sparse side information such as genetics may offer ideas for incorporating side-channel signals in bot scoring without destabilizing core behavioral models. The joint training across multiple groups (treatments) to improve prediction parallels training on heterogeneous user cohorts to enhance generalization. Thus, bot-defense practitioners could consider adopting EB-VAE style empirical Bayes priors and latent ODE decoders to fuse multimodal longitudinal metrics with discrete event timings for robust, interpretable detection models.

Cite

@article{arxiv2607_13984,
  title={ Multimodal Empirical Bayes Variational Autoencoders for Joint Longitudinal and Time-to-Event Modeling },
  author={ Anders Sjöberg and Nils Olsson and Marcus Baaz and Mats Jirstrand },
  journal={arXiv preprint arXiv:2607.13984},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.13984}
}

Read the full paper

• arXiv:2607.13984 (abstract)
• arXiv:2607.13984 (PDF)