Language-Based Digital Twins for Elderly Cognitive Assistance
Source: arXiv:2606.27334 · Published 2026-06-25 · By Mohammad Mehdi Hosseini, Mohammad H. Mahoor, Hiroko H. Dodge
TL;DR
This paper addresses the challenge of early detection and continuous monitoring of Mild Cognitive Impairment (MCI) in elderly individuals by proposing a novel language-based digital twin framework. The framework leverages large language models (LLMs) fine-tuned on longitudinal conversational data enriched with stylometric cues such as pause durations and speech tempo, alongside participant metadata. This approach aims to capture individual-specific linguistic style and temporal dynamics in naturalistic conversations to model cognitive health trajectories more holistically than conventional predictive methods.
The authors introduce a multi-head conditional variational autoencoder (cVAE) to jointly evaluate the linguistic fidelity of generated responses and predict cognitive scores (MoCA). Evaluations on the I-CONECT dataset, a longitudinal clinical trial corpus of elderly conversations, demonstrate that the digital twin closely reproduces identity-specific conversational patterns and achieves reconstruction and MoCA prediction errors comparable to real participant data. Furthermore, it outperforms baseline GPT-generated responses, indicating superior cognitive consistency. These results highlight the potential of language-based digital twins as scalable, non-invasive, and personalized tools for cognitive health monitoring in older adults.
Key findings
- Fine-tuned GPT-4.1-mini digital twin achieves identity classification accuracy (embedding Mean+STD features) of 44.15%, close to real data at 48.55%, versus 19.90% for base GPT (Table 1).
- Reconstruction error (MSE, range 0-1) for digital twin responses ranges from 0.0084 to 0.0098 across five participants, nearly matching real response errors of 0.0077 to 0.0094 (Table 2).
- MoCA score prediction error by cVAE is low for digital twin outputs, e.g., 0.55–1.08, close to real participant errors (0.40–1.05), but substantially worse for raw GPT (3.53–5.08) (Table 3).
- Stylometric features PAUSE and TEMPO, encoded as token annotations, enable the digital twin to capture temporal speech dynamics correlated with cognitive status.
- Data preprocessing included ASR correction with Whisper and speaker diarization with pyannote, yielding cleaner transcripts for fine-tuning.
- cVAE jointly optimizes reconstruction, latent KL divergence, and cognitive score regression losses, enabling integrated evaluation of linguistic fidelity and cognitive alignment.
- The digital twin model fine-tunes GPT-4.1-mini with system prompts conditioning on participant metadata and stylometric cues, facilitating personalized conversational behavior modeling.
- I-CONECT naturalistic conversational dataset of ~70 elderly participants aged 75+, with rich longitudinal dialogue recordings and MoCA scores, enables modeling temporal cognitive changes.
Threat model
Adversaries are passive observers interested in modeling or monitoring elderly individuals’ cognitive states through naturalistic language data. They possess access to conversational transcripts enriched with metadata but cannot manipulate inputs or directly query the model. Active attacks, model inversion, or adversarial input crafting are out of scope.
Methodology — deep read
The authors develop a language-based digital twin to mimic individual conversational behavior and cognitive state over time. The threat model assumes passive monitoring with access to naturalistic longitudinal conversational data enriched with contextual metadata and stylometric cues; adversaries do not have direct access to internal model states or private cognitive scores.
Data comes from the I-CONECT dataset, a clinical trial corpus including longitudinal conversations from about 70 older adults (aged 75+), both cognitively normal and with MCI. The study focuses on a subset of five participants with the most sessions for sufficient longitudinal data. Transcripts are cleaned by reprocessing audio via Whisper ASR and speaker diarization with pyannote to isolate participant utterances. This produces question-response pairs labeled with participant IDs, age, gender, interview date, and topics. Stylometric features PAUSE and TEMPO are tokenized into discrete categories to capture temporal aspects of speech.
The core digital twin model fine-tunes GPT-4.1-mini through supervised fine-tuning (SFT) to generate participant-specific responses conditioned on question text and metadata prompt inputs. The loss optimized is the cross-entropy log-likelihood of the target response given inputs. This allows the model to learn personalized linguistic style, semantic content, and temporal pacing encoded by stylometric tokens.
To jointly evaluate the digital twin’s fidelity and cognitive alignment, the authors propose a multi-head conditional variational autoencoder (cVAE) evaluator. The cVAE encoder takes question, metadata, and (generated or real) response as input, mapping them to a latent space parameterized by mean and log-variance. A decoder reconstructs the response, and an auxiliary head predicts the MoCA cognitive score. The total loss combines reconstruction MSE, KL divergence regularization, and MSE of cognitive prediction, weighted by a hyperparameter.
Training setup involves: processing batches of question-response pairs from selected participants, embedding textual features with Sentence-BERT, reducing dimensionality via PCA for topic descriptors, extracting sentiment, and including stylometric tokens. Training is conducted over multiple epochs until validation losses stabilize (e.g., Fig 2, Fig 3 show convergence). Hardware details and random seeds are not specified.
For evaluation, identity preservation is measured by applying an SVM classifier to embedding, sentiment, and combined features extracted from generated responses, assessing correctness of participant attribution. Fidelity is measured by cVAE reconstruction error (MSE between generated and real response features). Cognitive alignment is evaluated by MoCA score prediction error using cVAE outputs compared across real participant responses, base GPT generations, and the fine-tuned digital twin.
The evaluation estimates performance on held-out longitudinal sessions from the same participants but does not test cross-participant generalization or adversarial robustness explicitly. Code is provided on GitHub, with dataset available per clinical trial data sharing protocols.
An end-to-end example: given a question and metadata prompt for participant P3 aged 80 at the third interview, the fine-tuned model generates a response enriched with PAUSE and TEMPO tokens learned from that participant’s conversational history. The cVAE encoder processes the same question, metadata, and this generated response to produce a latent representation. The decoder attempts to reconstruct the original response, while the auxiliary head predicts a MoCA score close to that participant’s observed cognitive score. The reconstruction loss and cognitive prediction loss quantify how well the digital twin emulates P3’s linguistic style and cognitive state.
Technical innovations
- Integration of stylometric temporal speech features (PAUSE and TEMPO) as discrete tokens into LLM fine-tuning for personalized conversational behavior modeling.
- Use of a multi-head conditional variational autoencoder (cVAE) to jointly evaluate linguistic reconstruction fidelity and predict cognitive scores from generated text.
- Framework conditioning on participant-specific metadata alongside conversational input to enable individualized digital twin response generation.
- Demonstrated preservation of identity-specific linguistic patterns over longitudinal naturalistic conversations, aiding continuous cognitive status monitoring.
Datasets
- I-CONECT — ~70 elderly participants, naturalistic longitudinal conversation transcripts with metadata and MoCA cognitive scores — public clinical trial dataset
Baselines vs proposed
- Base GPT (raw generation): participant identity accuracy (embedding Mean+STD) = 19.90% vs digital twin = 44.15% vs real data = 48.55% (Table 1).
- Base GPT MoCA prediction error ranges 3.53–5.08 vs digital twin ranges 0.40–1.08 vs real participant error 0.40–1.05 (Table 3).
- Reconstruction MSE errors: real responses 0.0077–0.0094 vs digital twin 0.0084–0.0098 (Table 2).
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.27334.
Fig 1: Overview of the proposed language-based digital twin framework. The model
Fig 2 (page 2).
Fig 3 (page 2).
Fig 4 (page 2).
Fig 5 (page 2).
Fig 6 (page 2).
Fig 7 (page 2).
Fig 8 (page 2).
Limitations
- Small participant sample size (only five individuals with extensive longitudinal data) limits generalizability and robustness.
- Evaluation focuses on intra-participant longitudinal data; cross-participant generalization and transfer are not assessed.
- No explicit adversarial robustness or security threat testing against malicious input or model manipulation.
- Reliance on ASR and diarization preprocessing may introduce errors affecting digital twin fidelity.
- Stylometric tokens cover only pause and tempo, omitting other speech features such as prosody or acoustic qualities.
- MoCA prediction errors, while low, depend on the accuracy and variability of clinical cognitive assessments.
Open questions / follow-ons
- How well does the digital twin generalize to a larger and more diverse elderly population, including different languages and cultural backgrounds?
- Can the framework incorporate multimodal data beyond text, such as vocal prosody, facial expressions, and physiological signals for enhanced cognitive assessment?
- What is the robustness of the approach under noisy or adversarial conversational inputs common in real-world deployments?
- How can longitudinal digital twins be used to predict future cognitive decline or intervention effects beyond immediate MoCA scores?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, this work provides insight into sophisticated personalized language modeling of vulnerability-relevant populations, such as the elderly with cognitive impairments. The use of stylometric and contextual metadata features could inspire more fine-grained behavioral fingerprinting techniques beyond generic text analysis. However, the focus on longitudinal individual modeling presents privacy considerations and risks if applied to bot detection or user authentication without consent.
Furthermore, the introduced cVAE evaluator offers a novel joint metric combining behavioral fidelity and cognitive state prediction, which could be adapted for evaluating chatbot authenticity or detecting synthetic user simulations in security contexts. While this work does not address adversarial attacks or robustness directly, its methodology highlights how large language models can be fine-tuned to faithfully emulate subtle individual traits, posing challenges for bot-defense when confronting highly personalized conversational agents. Overall, the paper’s approach underscores the trade-offs in leveraging personalized language-based models for sensitive health monitoring versus security and privacy safeguards.
Cite
@article{arxiv2606_27334,
title={ Language-Based Digital Twins for Elderly Cognitive Assistance },
author={ Mohammad Mehdi Hosseini and Mohammad H. Mahoor and Hiroko H. Dodge },
journal={arXiv preprint arXiv:2606.27334},
year={ 2026 },
url={https://arxiv.org/abs/2606.27334}
}