Self-Evolving Human-Centered Framework for Explainable Depression Symptom Annotation

Source: arXiv:2607.15202 · Published 2026-07-16 · By Hoang-Loc Cao, Van Pham, Truong Thanh Hung Nguyen, Phuc Truong Loc Nguyen, Phuc Ho, Veronica Whitford et al.

TL;DR

This paper addresses the significant challenge of obtaining high-quality, explainable annotations for depression symptom datasets aligned with DSM-5-TR criteria. Current depression datasets often lack symptom-level justification and structured evidence, limiting transparency and interpretability in mental health AI systems. To overcome this, the authors propose a self-evolving, expert-in-the-loop annotation framework that combines LLM-assisted candidate evidence selection with expert verification and iterative feedback integration. The process operates in three stages: filtering evidence from clinical narratives, criterion-level symptom analysis aligned with DSM-5-TR, and case-level synthesis producing diagnostic and severity labels along with clinical rationales and audit trails. A dual-memory architecture (Example and Reflection Memories) enables incorporating expert feedback to improve future annotations without retraining the underlying model.

Key findings

• The framework achieves over 91% F1 score at sentence-level evidence detection across three LLM backbones (Gemini-3.5-Flash-Lite, GPT-4o-mini, GPT-5.4-mini).
• At the DSM-5-TR criterion level, GPT-4o-mini attains the highest F1 of 81.0%, with 82.9% precision, demonstrating effective symptom classification.
• Evidence-pair alignment (correct symptom linked to correct text) is more challenging, with best F1 at 67.0% by Gemini-3.5-Flash-Lite, highlighting the difficulty of explainable evidence grounding.
• Case-level MDD diagnosis accuracy reaches 90.0% for Gemini-3.5-Flash-Lite and GPT-5.4-mini, confirming reliable end-to-end annotation.
• AI-assisted labeling reduces expert annotation time by 63-75% compared to manual annotation, shifting experts from full annotation to verification.
• Gemini-3.5-Flash-Lite requires the fewest expert edits (10.2 per case) and evidence edits (6.5 per case), indicating better evidence quality and usability.
• GPT-4o-mini results in the fewest symptom criterion flips (1.6 per case), aligning with strongest criterion-level classification metrics.
• No model alone dominates all metrics; complementary strengths suggest the framework is robust across LLM backbones.

Threat model

The adversary is assumed to be the natural challenges in annotating complex clinical depression narratives, including ambiguous symptom expressions and inconsistent expert judgments. The framework assumes trustworthy human experts verify and correct AI outputs. It does not address malicious adversarial attacks on the annotation process or model manipulation.

Methodology — deep read

The research proposes a collaborative annotation framework supporting explainable depression symptom labeling aligned with DSM-5-TR criteria. The threat model involves clinical experts acting as human verifiers working alongside AI-generated draft annotations; the adversary model is not strictly adversarial but rather aims to minimize expert cognitive burden and errors.

The dataset comprises 10 complex clinical cases from the public ReDSM5 Reddit-based depression benchmark dataset, annotated independently by 5 expert psychologists following DSM-5-TR. Consensus gold standard labels cover sentence-level evidence, symptom criteria (A1-A9), and final MDD diagnosis.

The annotation pipeline has three stages: (1) Evidence-based Screening where LLMs extract candidate symptom-relevant sentences from raw clinical text, using contextual retrieval from Example Memory; (2) Criterion-Level DSM-5-TR Analysis where filtered text is mapped to the nine MDD DSM-5-TR criteria, generating structured criterion properties including binary or categorical symptom judgments, clinical rationales, supporting evidence quotes, and highlighted phrases; and (3) Self-Evolving Case Export where these refined criteria synthesize an overall MDD diagnosis and severity rating, alongside a clinical summary.

A core innovation is the dual-memory architecture enabling self-evolution: Example Memory stores expert-verified gold cases as few-shot examples, and Reflection Memory distills recurrent expert correction patterns into rules or insights. These memories update iteratively with expert corrections but do not update model parameters, enabling prompt-level adaptation in future annotation tasks.

The system was implemented using three LLM backbones (Gemini-3.5-Flash-Lite, GPT-4o-mini, and GPT-5.4-mini). Autonomous AI annotations were collected before expert review to evaluate intrinsic AI quality.

Evaluation protocols measured precision, recall, and F1-score at sentence-level evidence detection, DSM-5-TR criterion classification, evidence-to-criterion alignment, and final case-level diagnosis accuracy. Expert revision effort was measured by time savings, total edits, criterion flips, and evidence edits comparing manual vs AI-assisted annotation.

A concrete example: a clinical narrative is segmented into sentences, candidate symptom-relevant sentences retrieved, expert verifies and edits these. Then each sentence’s alignment with DSM-5-TR criteria is assessed with generated rationales and highlights. The expert adjusts criterion labels aided by conflict warnings for ambiguous signs. Finally, the diagnosis and severity are proposed, the expert approves, and the case plus expert corrections are stored in dual memory. Future annotation tasks leverage this memory for improved prompt retrieval and output quality.

The framework emphasizes explainability, human-centered design with conflict warnings and interactive GUI, and auditability through preservation of evidence and revision histories. The pilot study was small scale but provided detailed multi-level evaluation of annotation quality and human workload reduction. The authors note that cross-validation on larger datasets and longitudinal memory updates remain to be studied.

Technical innovations

• Integration of a three-stage collaborative pipeline combining evidence screening, criterion-level DSM-5-TR symptom analysis, and case-level diagnosis with expert-in-the-loop verification.
• Introduction of a dual-memory architecture (Example and Reflection Memories) that internalizes expert feedback and adapts annotation behavior without retraining model parameters.
• Development of a structured annotation scheme exporting final labels alongside supporting clinical evidence, clinical rationales, key phrase highlights, and complete audit trails for explainability and auditability.
• Use of conflict warning mechanisms at the criterion annotation stage to alert experts of ambiguous evidence, enhancing clinical coherence and expert control.

Datasets

• ReDSM5 — 10 complex clinical Reddit cases with gold DSM-5-TR symptom and diagnosis labels — publicly available research benchmark

Baselines vs proposed

• Gemini-3.5-Flash-Lite: Sentence-level F1 = 93.8% vs GPT-4o-mini: 91.3%, GPT-5.4-mini: 93.6%
• GPT-4o-mini: Criterion-level F1 = 81.0% vs Gemini-3.5-Flash-Lite: 76.1%, GPT-5.4-mini: 77.3%
• Gemini-3.5-Flash-Lite: Evidence-pair F1 = 67.0% vs GPT-4o-mini: 66.7%, GPT-5.4-mini: 57.1%
• Gemini-3.5-Flash-Lite & GPT-5.4-mini: Case-level diagnosis accuracy = 90.0% vs GPT-4o-mini: 80.0%
• Time saved by annotation workflow: Gemini-3.5-Flash-Lite 75.0% vs GPT-4o-mini 70.0%, GPT-5.4-mini 63.0%
• Total expert edits per case: Gemini-3.5-Flash-Lite 10.2 vs GPT-4o-mini 10.4, GPT-5.4-mini 12.9

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.15202.

Fig 1: Overview of the proposed human-centered, self-evolving DSM-5-TR annotation framework. The pipeline consists of three stages: (1) screening and

Fig 2: Expert annotation interface for reviewing AI-suggested DSM-5-TR evidence, highlighted clinical cues, and criterion-level labels.

Limitations

• Pilot study limited to 10 cases, constraining statistical power and generalizability.
• Evaluation did not include longitudinal assessment of memory updates or multiple feedback cycles.
• The framework focuses on annotation quality, not on automating clinical diagnosis or real-world deployment.
• The dual-memory self-evolution approach was described but its effectiveness remains empirically untested beyond initial pilot.
• LLM backbones used are proprietary or recently released; impact of different model families or smaller models was not studied.
• No adversarial robustness or out-of-distribution generalization evaluation was conducted.

Open questions / follow-ons

• How effectively does the dual-memory structure improve future annotation quality over multiple feedback iterations without retraining?
• Can the framework scale to larger and more diverse clinical datasets, including other mental health disorders beyond depression?
• How robust is the system under noisy or incomplete clinical text inputs or in cross-domain settings?
• What are best practices for integrating conflicting expert feedback into the self-evolving memory architecture?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners focused on human-AI collaboration and explainability, this study highlights the importance of structured annotation workflows incorporating human verification alongside AI assistance. The dual-memory self-evolving approach offers a conceptual model for iterative improvement of annotation or labeling tasks without full model retraining, which could inspire adaptive bot detection calibration pipelines. Furthermore, its emphasis on multi-level explainability, traceable evidence linking, and audit trails aligns with key bot defense needs for robust interpretability and transparent decision-making. The conflict warning mechanism may motivate analogous alerting for ambiguous or adversarial inputs in security-critical systems. Though domain-specific to clinical text, the principles of human-centered iterative annotation with AI assistance apply broadly to improving label quality and reducing human effort in security and bot detection tasks.

Cite

@article{arxiv2607_15202,
  title={ Self-Evolving Human-Centered Framework for Explainable Depression Symptom Annotation },
  author={ Hoang-Loc Cao and Van Pham and Truong Thanh Hung Nguyen and Phuc Truong Loc Nguyen and Phuc Ho and Veronica Whitford and Hung Cao },
  journal={arXiv preprint arXiv:2607.15202},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.15202}
}

Read the full paper

• arXiv:2607.15202 (abstract)
• arXiv:2607.15202 (PDF)