Guiding Language Models to Be More Empathetic: Culturally Sensitive Mental Health Advice Generation Through Human-LLM Collaboration

Source: arXiv:2607.23538 · Published 2026-07-26 · By Fatema Tuj Johora Faria, Mukaffi Bin Moin, Md. Mahfuzur Rahman, Khan Md Hasib, Jubayer Al Mahmud, M. F. Mridha

TL;DR

This paper addresses the challenge of generating empathetic and culturally sensitive mental health counseling responses in Bangla, a low-resource language with limited prior research on response generation. The authors curate a novel dataset of 625 authentic Bangla mental health cases collected from Facebook posts, a popular Bangladeshi TV program, and student questionnaires. They assemble a gold standard of expert-authored counseling responses from licensed clinical psychologists and compare responses generated by three proprietary large language models (LLMs): GPT-4o Mini, Claude 4.5 Haiku, and Gemini 2.5 Pro. To improve LLM output quality, they propose RP-RCAF, a novel role-playing reflective chain-of-thought prompting framework combining expert few-shot examples with structured self-reflection, aimed at eliciting compassionate, culturally grounded, and ethically aligned advice. They also develop an evaluation framework (G-REFS) that integrates automated scoring via a Grok 4 LLM judge with expert psychologist validation across emotional sensitivity, cultural appropriateness, linguistic clarity, and ethical soundness. Experimental results demonstrate that RP-RCAF consistently outperforms standard zero-shot and few-shot prompting baselines for all tested LLMs, producing responses closer to licensed human expert quality particularly in emotional and ethical dimensions. However, culturally nuanced and high-risk scenarios (e.g., sexual abuse, self-harm ideation) still require human-in-the-loop oversight to ensure safety. Among models, Gemini 2.5 Pro shows the strongest overall performance. This work establishes a valuable benchmark and methodological foundation for culturally sensitive AI mental health support in Bangla and similar low-resource languages.

Key findings

• RP-RCAF framework increases average G-REFS scores by 16.67%–32.26% over zero-shot prompting and 7.69%–12.50% over few-shot prompting across T1 (emotional sensitivity), T2 (cultural appropriateness), T3 (linguistic clarity), and T4 (ethical soundness) metrics.
• Gemini 2.5 Pro achieves the highest overall average G-REFS score of 4.2 compared to Claude 4.5 Haiku (4.0) and GPT-4o Mini (3.9) across eight major mental health categories.
• LLM responses perform competitively with human expert responses on linguistic clarity (T3, average ~4.2 vs 4.8 for humans) but lag notably on cultural appropriateness (T2) and ethical soundness (T4) especially in sensitive categories like sexual abuse and self-destruction.
• Human-in-the-loop revision improves quality scores of borderline responses (average score 2.5–4), increasing proportion of accepted responses from initial LLM output by approximately 20–30%.
• Moderate agreement (overall ICC = 0.71) between the Grok 4 automated evaluation and licensed psychologist ratings validates G-REFS as a reliable automated quality assessment tool for non-high-risk counseling cases.
• Lowest G-REFS and human rating agreement observed in high-risk cases (sexual abuse ICC=0.59, self-destruction ICC=0.57) indicating that expert oversight remains crucial for these contexts.
• Across mental health categories, T2 (cultural appropriateness) is consistently the weakest scoring dimension for LLM responses, highlighting a persistent challenge in encoding sociocultural nuance.
• Expert-authored responses maintain the highest scoring baseline with mean scores ~4.7–4.8 across all dimensions, setting a strong target for AI-generated counseling.

Threat model

The adversary is implicit in the design: the system must prevent generation of harmful, unethical, or culturally insensitive mental health advice. The models are instructed not to provide clinical diagnoses or prescriptions. The threat is mitigated via prompting constraints and expert oversight, assuming adversaries do not have direct control over the model weights or prompt inputs. There is no explicit adversarial attack evaluation.

Methodology — deep read

The study begins with a clearly defined threat model assuming the task of generating culturally sensitive, empathetic, and ethically sound mental health advice for Bangla speakers, focusing on emotional support rather than clinical diagnosis or medical intervention. The adversary is not explicitly modelled as malicious, but the framework incorporates ethical safeguards to avoid harmful outputs.

Data collection involved curating 625 authentic mental health counseling cases sourced from (1) publicly available Facebook posts (filtered and translated to standard Bangla using GAMITISA tools), (2) transcripts of the Bangladeshi TV program “Ami Akhon Ki Korbo”, and (3) anonymized voluntary student questionnaire responses. All data underwent rigorous anonymization removing personally identifiable information and normalization to preserve dialectal and emotional content. Cases were categorized into mental health topics relevant to the Bangladeshi context such as academic stress, self-harm ideation, sexual abuse, and family conflict.

Licensed clinical psychologists authored empathetic, ethical, and culturally nuanced counseling advice for ~30% of the cases to serve as gold-standard references. These experts independently assessed responses, with high inter-rater agreement (Cohen's kappa = 0.83). LLM-generated responses were produced for the remaining ~70% cases using three proprietary LLMs: GPT-4o Mini, Claude 4.5 Haiku, and Gemini 2.5 Pro.

The core technical contribution is the Role-Playing Reflective Chain-of-Thought Advisory Framework (RP-RCAF), a few-shot prompting strategy where LLMs assume the persona of a compassionate Bangladeshi mental health advisor. The prompt combines expert-authored few-shot examples with structured intermediate reasoning steps (chain-of-thought) that guide the model to consider emotional state, cultural context, ethical constraints, and actionable advice before generating final responses. This is designed to produce emotionally supportive, culturally grounded, and ethically responsible counseling advice.

LLM parameters were standardized (temperature=0.7, top-p=0.9). Multiple response variants per query were generated and stored. All responses were initially scored by Grok 4, an LLM evaluator in the Grok 4-Based Response Evaluation and Scoring Framework (G-REFS), which rates four dimensions: emotional sensitivity (T1), cultural appropriateness (T2), linguistic clarity (T3), and ethical soundness (T4), each on a 5-point Likert scale with detailed rubrics. Average scores determined categorization into accept (≥4), revise (2.5–4), or reject (<2.5).

A human-in-the-loop review involved two licensed clinical psychologists who further reviewed accepted responses and supervised expert annotators revising borderline responses to improve emotional tone, cultural relevance, and ethical safety. Revised responses were re-scored by Grok 4 for final inclusion.

Evaluation involved quantitative metrics, category-wise analysis, comparison of zero-shot, few-shot, and RP-RCAF prompting methods, inter-rater agreement analysis using ICC between Grok 4 and human raters, and qualitative inspection of counseling content. The pipeline supports single-turn counseling cases. Code and data release status is not specified, but the dataset MindSpeak-Bangla is constructed for this work with some public data sources. Overall, the methodology tightly integrates data curation, expert human annotation, novel prompting, LLM response generation, automated and expert evaluation, and iterative refinement to establish a practical framework for AI mental health counseling in a low-resource language context.

Technical innovations

• Role-Playing Reflective Chain-of-Thought Advisory Framework (RP-RCAF): A few-shot prompting strategy combining expert examples with intermediate reflective reasoning to guide empathetic, culturally aware, and ethical mental health response generation.
• Grok 4-Based Response Evaluation and Scoring Framework (G-REFS): An automated LLM-based evaluation framework with psychologist validation to quantitatively assess emotional sensitivity, cultural appropriateness, linguistic clarity, and ethical soundness of counseling responses.
• A curated MindSpeak-Bangla dataset of 625 authentic Bangla mental health counseling cases with expert and LLM-generated responses, enabling culturally grounded AI research for a low-resource language.
• Integration of human-in-the-loop expert review and editing processes to refine and validate LLM-generated mental health counseling responses, especially in high-risk scenarios.

Datasets

• MindSpeak-Bangla — 625 cases — collected from Bangladeshi Facebook posts, TV transcripts, and student questionnaires (anonymized)
• Expert-authored counseling responses — ~188 cases — licensed clinical psychologists in Bangladesh

Baselines vs proposed

• GPT-4o Mini zero-shot prompting: average G-REFS score T1=3.0, T2=2.8, T3=4.0, T4=3.2 vs RP-RCAF prompting: T1=3.9 (+30%), T2=3.6 (+28.57%), T3=4.2 (+5%), T4=3.9 (+21.88%)
• Claude 4.5 Haiku zero-shot prompting: T1=3.1, T2=2.9, T3=4.1, T4=3.3 vs RP-RCAF prompting: T1=4.1 (+32.26%), T2=3.8 (+31.03%), T3=4.4 (+7.32%), T4=4.0 (+21.21%)
• Gemini 2.5 Pro zero-shot prompting: T1=3.6, T2=3.2, T3=4.2, T4=3.5 vs RP-RCAF prompting: T1=4.2 (+16.67%), T2=3.9 (+21.88%), T3=4.5 (+7.14%), T4=4.2 (+20%)
• Expert human responses average across categories: T1=4.7, T2=4.6, T3=4.8, T4=4.7, outperforming all LLMs
• Inter-rater reliability ICC between G-REFS and human experts = 0.71 overall; category-wise ICC as low as 0.57 for self-destruction cases, highest for loneliness (~0.75)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.23538.

Fig 1: Overview of the Bangla mental health counseling case study. (1) Authentic counseling cases are collected

Fig 2: Overview of the proposed RP-RCAF and G-REFS frameworks. (1) RP-RCAF employs role-playing and

Fig 3 (page 3).

Fig 4 (page 3).

Fig 5 (page 3).

Fig 6 (page 3).

Fig 7 (page 3).

Fig 8 (page 5).

Limitations

• Focuses on nonclinical mental health counseling for emotional support, not diagnosis or medical treatment.
• Dataset derived from Facebook, TV, and student questionnaires may not fully represent socioeconomically and demographically diverse Bangladeshi populations.
• RP-RCAF relies heavily on prompt engineering, making reproducibility and transferability to other frameworks uncertain.
• High-risk topics like sexual abuse and self-harm still require expert human oversight due to safety and ethical concerns.
• Only single-turn response scenarios were considered; lacks modeling for long-term conversational context in real counseling.
• Evaluations limited to proprietary LLMs; results may not generalize to open-source or future models.
• Potential residual ethical and privacy risks despite anonymization and expert review of sensitive data.

Open questions / follow-ons

• How to extend the RP-RCAF and evaluation frameworks for multi-turn, long-term mental health counseling conversations?
• Can the approach be generalized to other low-resource languages with distinct cultural norms and mental health needs?
• How to improve automated evaluation (G-REFS) agreement with human experts specifically in high-risk and culturally complex scenarios?
• What methods can increase the robustness and reproducibility of prompt engineering strategies like RP-RCAF across diverse LLM architectures?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners working on mental health chatbot systems, this study offers important insights into how culturally grounded prompt engineering can significantly enhance the empathy, safety, and cultural appropriateness of LLM-generated counseling responses beyond standard prompting methods. The introduction of a chain-of-thought reflective approach (RP-RCAF) guided by expert examples could be adapted to improve dialog agents tasked with sensitive support conversations, ensuring alignment with ethical guidelines and nuanced cultural contexts. The hybrid automated plus human expert validation framework (G-REFS) establishes a scalable quality assurance mechanism that could inspire similar multi-stage evaluation pipelines in safety-critical AI deployments. However, practitioners must note the limitations, including the necessity of human-in-the-loop review for high-risk cases and the sensitivity of response quality to prompt design. Additionally, the single-turn focus and proprietary model reliance suggest the need for further research before deployment in real-world mental health systems, especially those serving diverse populations within captcha-protected services or mental health triage bots.

Cite

@article{arxiv2607_23538,
  title={ Guiding Language Models to Be More Empathetic: Culturally Sensitive Mental Health Advice Generation Through Human-LLM Collaboration },
  author={ Fatema Tuj Johora Faria and Mukaffi Bin Moin and Md. Mahfuzur Rahman and Khan Md Hasib and Jubayer Al Mahmud and M. F. Mridha },
  journal={arXiv preprint arXiv:2607.23538},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.23538}
}

Read the full paper

• arXiv:2607.23538 (abstract)
• arXiv:2607.23538 (PDF)