MAM-AI: An On-Device Medical Retrieval-Augmented Generation System for Nurses and Midwives in Zanzibar

Source: arXiv:2606.29580 · Published 2026-06-28 · By Yi Ren

TL;DR

This paper presents MAM-AI, an on-device medical retrieval-augmented generation (RAG) system designed specifically for nurse-midwives in Zanzibar, where connectivity is intermittent and clinical guidelines are lengthy. The system operates fully offline on commodity Android hardware by embedding user questions using a 300M-parameter model (EmbeddingGemma) to retrieve relevant passages from a curated corpus of 87 authoritative clinical guideline documents (63,650 passages). These passages are then used by a 4B-parameter int4 quantized language model (Gemma 4 E4B) to generate cited, clinically relevant answers. The evaluation employs a layered methodology including retriever and generator isolated assessments, end-to-end tests on two purpose-built benchmarks with LLM judges validated against physician rubrics, and latency measurements.

Results show that on-device retrieval is essentially solved, with the 300M embedder ranking 3rd out of 7 retrievers and comparable to cloud systems. The main limitation is the generator; adding retrieved context does not improve answer quality and can worsen performance for larger models. Among two candidate 4B generators of the same size but different safety profiles, the deployed model was the safer one that was about twice as faithful to its sources, though less helpful, with helpfulness improved via a redesigned system prompt that reduces answer deflection from 33% to 3%. Corpus quality critically impacts specificity and actionability of answers. The system is open source and fully evaluated on its exact deployed configuration, but remains a research prototype without a field trial.

Key findings

• On-device retrieval using EmbeddingGemma (300M parameters) ranks third out of seven retrievers evaluated on mamaretrieval, delivering retrieval quality close to stronger cloud-based systems with single-digit percentage points gap.
• Adding retrieved context to the 4B int4 quantized generator typically does not improve and sometimes harms answer quality, as measured by key-fact recall and rubric scores.
• Two same-size 4B candidate generators exhibit a trade-off between helpfulness and safety: the more helpful model commits numerous genuine dangerous clinical errors (e.g., 4, 11, and 15 dangerous answers across prompts), while the safer model incurs about twice the faithfulness to source passages and near-zero genuine dangerous answers.
• A redesigned system prompt (G1) cuts answer deflection—where the model declines to provide clinical guidance—from ~33% to ~3%, roughly doubling key-fact recall and improving rubric scores with minimal increase in unsafe answers.
• Generator capability (model size) is the dominant lever on overall quality: evaluation across models from 2B to 397B parameters shows key-fact recall and rubric completeness rise steeply up to ~30B parameters, then plateau, while safety improves monotonically with size.
• Quantization precision from int4 to bf16 has negligible effect on accuracy and safety for the deployed 4B model, indicating int4 quantization is efficient with little performance sacrifice.
• Corpus quality is decisive: when the corpus contains relevant passages, answers are specific and actionable; when relevant material is missing, answers are vague or omitted.
• Latency on a current Android device allows cited answers in tens of seconds fully offline, with performance degrading on lower-end hardware.

Threat model

The system assumes deployment in intermittently connected, low-resource clinical settings where no network connectivity is available after initial setup. The main safety concern is preventing the model from providing harmful or dangerous medical advice, especially given that nurse-midwives have variable midwifery training. No adversarial user is assumed, and the system is designed to not transmit any patient or query data off-device, protecting privacy and mitigating network attack vectors. The adversary cannot access the device runtime or tamper with the guideline corpus bundles.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is not explicitly defined as a malicious actor but the system focuses strongly on safety to avoid harmful clinical advice that may cause direct patient harm. The system assumes no network connectivity post-setup, thus no patient-related queries leave the device, protecting privacy and precluding adversarial network attacks. It aims to assist nurse-midwives with variable training by providing authoritative, evidence-based guideline-driven assistance.

2. Data: The curated knowledge base consists of 87 authoritative clinical guideline documents relevant to maternal, newborn, obstetrics, and reproduction health. These include WHO guidelines, UK NICE, professional midwifery college standards, field references for low-resource settings, and Zanzibar/Tanzania national policies. In total, the corpus contains 63,650 text passages chunked by document structure (headings). The corpus is versioned and each chunk is hashed for traceability. Data preprocessing uses an ML layout parser to extract structured text from PDFs, followed by embedding passage text with EmbeddingGemma 300M.

3. Architecture / Algorithm: The system is a retrieval-augmented generation (RAG) pipeline running entirely on-device. Queries are embedded with EmbeddingGemma-300M to retrieve top-3 passages from the SQLite vector store of embedded corpus passages. These retrieved passages are added to the prompt given to the generator. The generator is Gemma 4 E4B, a 4B parameter int4 quantized language model running inference on the device. The model generates a clinical answer citing specific source passages inline. The deployed system prompt ("G1") is carefully designed to reduce deflection and encourage helping behavior while preserving clinical safety. Generation is done on CPU by default, with optional GPU execution if available.

4. Training Regime: The model weights are off-the-shelf open models from Google, loaded int4 quantized. The paper does not retrain or fine-tune models. Instead, the main contributions are system design, corpus curation, prompt engineering, and evaluation. Evaluation is performed on a GPU cluster as a proxy for on-device generation.

5. Evaluation Protocol: Layered evaluation includes end-to-end system tests, retriever-only evaluation on mamaretrieval (3,185 queries with graded relevance), generator oracle-context evaluation, and latency measurements on actual Android hardware. End-to-end evaluation uses two purpose-built benchmarks: mamabench with Kenya Clinical Vignettes (312 cases) and HealthBench-oss (1209 cases). Answer scoring uses three metrics: key-fact recall against expert gold answers, safety ratings into four levels (safe, minor concern, potentially harmful, dangerous), and HealthBench rubric scores (weighted met criteria). An LLM judge (gpt-oss-120b) calibrated and validated against physician rubrics scores all answers. The LLM judge provides reproducible, open scoring closely matching human expert consensus. Safety adjudication includes a second automated expert review of dangerous cases.

6. Reproducibility: All code, knowledge base, benchmarks, and evaluation harness are publicly released as open source. The exact deployed configuration and model weights are specified and a browser demo runnable in Hugging Face Spaces is provided. Benchmarks mamabench and mamaretrieval come with construction code for replication. While the generator and embedder models are off-the-shelf, the full pipeline and evaluation are fully reproducible excluding proprietary hardware and data that require replicating document sources.

Concrete Example End-to-End: A nurse-midwife asks a clinical question on the Android device. The question is embedded using EmbeddingGemma-300M and matched via nearest neighbor in the SQLite vector store of 63,650 guideline passages. Top-3 passages are retrieved and concatenated with the prompt, which instructs the Gemma 4 E4B int4 quantized generator how to respond with citations. The generator produces a streamed answer citing passages inline. The entire process runs offline on-device with answer latency on the order of tens of seconds. The response is then scored with key-fact recall and safety judged by the LLM scoring system. This precise evaluation pipeline guides prompt and model choices.

Technical innovations

• Full offline, on-device RAG pipeline optimized for medical question answering with a 300M parameter embedder and 4B int4 quantized generator running on commodity Android devices.
• A curated, versioned clinical knowledge base of 87 guideline documents chunked into 63,650 passages and stored in an on-device SQLite vector store with stable citation keys for inline referencing.
• Design of a custom system prompt that drastically reduces model deflection from ~33% to ~3% while doubling answer completeness and maintaining safety.
• Layered, bottom-up evaluation methodology including an LLM judge rigorously validated against physician rubrics for reproducible safety and quality assessment in a low-resource setting context.

Datasets

• mamabench — 25,949 clinical QA items, open-ended and multiple-choice tracks (Kenya Clinical Vignettes subset: 312 cases; HealthBench-oss: 1,209 cases) — open source
• mamaretrieval — 3,185 clinical queries with graded relevance labels over the MAM-AI guideline corpus — open source
• MAM-AI knowledge base — 87 clinical guideline documents, 63,650 extracted passages (~260 MB) — publicly released curated corpus

Baselines vs proposed

• Retriever baselines: EmbeddingGemma-300M ranks 3rd/7 on mamaretrieval benchmark, close to best cloud retrievers.
• Generator baseline comparison: Gemma 4 E4B with G1 prompt vs Gemma 3n E4B (same size) — key-fact recall 0.279 vs 0.351; potentially harmful rate 15.7% vs 31.4%; dangerous answers 1 vs 11; chosen model trades helpfulness for safety.
• Prompt impact: Deflection cut from 32.7% to 3.2% on Kenya vignettes with G1 prompt on Gemma 4 E4B; key-fact recall doubled (0.139 to 0.279).
• Generator size ladder (no retrieval) on Kenya benchmark: recall rises from 0.27 (2B) to 0.47 (~30B), safety harm rate decreases from 26% (2B) to 6% (26–31B).
• Quantization effect at 4B model size negligible: int4 recall 0.328 vs Q8 0.331 and bf16 0.320 on Kenya benchmark.

Limitations

• No real-world field deployment or live user study due to development cycle constraints; all evaluation is on pre-collected benchmark data and synthetic LLM-judged assessments.
• Corpus coverage is fixed and limited to 87 documents; missing relevant clinical content causes vague or incomplete answers.
• Lack of multilingual prompt evaluation; bilingual English/Swahili support was withdrawn due to safety concerns with unvalidated language prompts.
• Safety adjudication depends on LLM automated reviewer with no independent clinical expert validation, potentially missing nuanced clinical risk or subtle errors.
• On-device latency is tens of seconds on modern phones but would be slower on lower-end hardware, impacting real-time usability.
• The generator cannot simultaneously maximize helpfulness and safety at 4B scale, necessitating hard tradeoffs in model selection.

Open questions / follow-ons

• How would MAM-AI perform and be received in real-world clinical deployments with nurse-midwives, including user acceptance and impact on patient outcomes?
• Can corpus coverage be expanded and dynamically updated while preserving offline operation, traceability, and safety?
• What methods could improve the helpfulness-safety tradeoff in small on-device generators, e.g., via constrained decoding, post-hoc verification or hybrid heuristics?
• How can multilingual support including Swahili be validated and integrated safely, given the local language requirements?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this work demonstrates a practical approach to deploying complex ML pipelines fully on-device under constrained connectivity and hardware scenarios—a key challenge for distributed, privacy-sensitive applications. The layered evaluation methodology, including use of validated LLM judges calibrated to human expert rubrics, provides a template for rigorous quality and safety assessment of ML models in-the-wild. The insights on generator prompt engineering to reduce evasive or deflective model responses while maintaining safety have parallels in security-sensitive conversational systems.

While not a bot-detection paper per se, MAM-AI's offline retrieval-augmented architecture, careful corpus curation, and generator faithfulness monitoring offer guiding principles for any application requiring trustworthy, domain-specific knowledge grounding and real-time response on end-user devices without server connectivity. The tradeoffs between model size, helpfulness, and safety echo broader challenges in deploying constrained language models for trustworthy AI tasks.

Cite

@article{arxiv2606_29580,
  title={ MAM-AI: An On-Device Medical Retrieval-Augmented Generation System for Nurses and Midwives in Zanzibar },
  author={ Yi Ren },
  journal={arXiv preprint arXiv:2606.29580},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.29580}
}

Read the full paper

• arXiv:2606.29580 (abstract)
• arXiv:2606.29580 (PDF)