ScaFE: Data-Efficient Scar Classification with LLM-Generated Clinical Feature Programs

Source: arXiv:2607.28538 · Published 2026-07-30 · By Ruman Wang, Hangting Ye

TL;DR

This paper addresses the challenge of classifying pathological scars—specifically, distinguishing keloids (KD) from hypertrophic scars (HS)—using clinical photographs collected from multiple hospitals with limited labeled data and large variation in image acquisition. Traditional end-to-end image classification models require substantial labeled data and are susceptible to domain shifts across sites. Moreover, directly using hosted vision-language models (VLMs) is problematic due to patient privacy, governance concerns, and opaque decision-making. The authors propose ScaFE (Scar Feature Engineering), a novel framework that leverages large language models (LLMs) not to diagnose images directly but to synthesize deterministic, executable feature programs representing clinically meaningful scar attributes. These feature programs execute locally on scar images to extract structured, interpretable features grounded in clinical evidence retrieved by the LLM. A lightweight Random Forest then classifies scars based on these features.

The key novelty is this separation of roles: the LLM incorporates clinical domain knowledge to generate auditable, evidence-backed feature extraction code; local execution ensures no raw image data leave the site; and a classic machine learning classifier produces predictions, enhancing auditability and robustness. On a leave-one-site-out evaluation of 600 clinical scar photos from three hospitals, ScaFE achieves 81.0% site-macro balanced accuracy, exceeding the best baseline (BiomedCLIP) by 10 points. It maintains a strong lead even when trained on only 10% of data, demonstrating data efficiency. Iterative refinement guided by aggregate validation metrics and SHAP feature importances improves program executability from 66.7% to 95%, and ensures 100% evidence-backed feature coverage. The authors thus show that clinical knowledge embedded in LLMs can be effectively transferred into reproducible, local, and interpretable medical image classification pipelines without direct image access by the LLM, improving cross-site generalization and auditability.

Key findings

• ScaFE achieves 81.0% site-macro balanced accuracy in leave-one-site-out evaluation on 600 images from three hospitals, exceeding BiomedCLIP by 10.0 percentage points (95% CI 7.2–12.8, p < 0.001).
• Using only 10% of the development data, ScaFE retains 72.0% balanced accuracy, maintaining an 11.8-point lead over BiomedCLIP.
• Iterative refinement increases executable feature program rate from 66.7% to 95.0%.
• Candidate features passing evidence verification rise from 70.0% to 91.7%, and final filtering ensures 100% coverage of clinically supported features.
• Ablation studies show removing online literature grounding reduces balanced accuracy by 4.9 points; removing prediction-level feedback loses 6.6 points; removing SHAP feature feedback loses 2.8 points; shuffling feedback destroys semantic guidance and reduces accuracy to 72.9%.
• Random Forest downstream learner outperforms logistic regression, SVM, single tree, and XGBoost baselines in accuracy and stability.
• SHAP feature importance ranking correlates strongly with feature utility—removing top-3 causes 8.6-point accuracy drop vs 2.1 for random deletion (p < 0.001).
• Local feature extraction runtime averages 34.7 ms per image, while the entire program search and refinement takes about 18.6 minutes per fold.

Threat model

The adversary is assumed unable to access raw clinical images or patient-level feature outputs, which remain confined locally. The LLM does not receive any patient or image data directly, only aggregate validation metrics and literature evidence. The threat model excludes attackers who can manipulate local image acquisition or execution environments but protects privacy by construction against remote or hosted LLM exposure of sensitive data. The system also prevents leakage through intermediate artifacts by sandboxing and evidence verification.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is implicitly any entity trying to breach patient privacy or manipulate scar classification. The design assumes that raw clinical images cannot leave local environments due to governance constraints, and the LLM has no direct access to raw images or patient-level data. The LLM operates only on aggregate validation metrics and clinical evidence retrieved from public literature. This prevents privacy leaks and ensures reproducibility. The adversary cannot access individual images or extracted feature vectors, only aggregated outputs.

2. Data: The dataset consists of 600 de-identified clinical scar photographs from three hospitals, each contributing 200 images balanced with 100 keloid and 100 hypertrophic scar cases. One image per patient was included, excluding duplicates. Reference diagnoses were verified by expert plastic surgeons with high agreement (Cohen's kappa 0.86). Photo acquisition was heterogeneous (72% smartphone, 28% DSLR) over 2020–2025. Data splits used leave-one-site-out cross-validation with an 80/20 split of development data for fitting and validation. Image data was resized to 512x512 for ScaFE; no lesion masks or metadata were provided.

3. Architecture / Algorithm: ScaFE decomposes the predictor into two parts: (a) an executable feature program g that maps images to K-dimensional interpretable clinical features (e.g., color, texture, morphology) and (b) a Random Forest classifier hg operating on these features. An LLM is used in T=3 iterative rounds to (1) autonomously retrieve clinical literature via web search, (2) generate Python feature extraction programs that encode clinically grounded scar attributes with traceable evidence citations, and (3) refine these programs using aggregate feedback metrics including balanced accuracy, class-wise confusion matrices, fractions of invalid/constant feature values, and SHAP feature importances. Candidate programs must execute locally in sandboxed environments with restricted libraries and fixed resources without network/file access.

4. Training Regime: At each round, M=4 candidate feature programs are generated and executed locally on the Dfit and Dval subsets. The RF classifier is fixed with 500 trees, max depth 8, and minimum leaf size 3. The RF is trained on Dfit features and evaluated on Dval to provide metrics for feedback to the LLM. No hyperparameter tuning of RF was performed during search to isolate representation effects. After the search, the best program g* is selected by highest validation balanced accuracy and frozen. Final RF model h* is retrained on the full development data and evaluated once on the sealed test set.

5. Evaluation Protocol: The primary metric is site-macro balanced accuracy (BAcc), averaged equally across the three hospitals. Other metrics include macro-F1, AUROC, Brier score, calibration error, sensitivity, and specificity. Controls include shuffled label permutations and random feature deletions. Multiple baseline methods were compared including handcrafted clinical features with RF, fine-tuned CNNs (ResNet-18, EfficientNet-B0), frozen foundation models with linear probes (DI-NOv3, Derm Foundation, BiomedCLIP), and a local VLM-direct approach. Five random seeds with bootstrapped 95% confidence intervals were reported. Ablations tested the contribution of online evidence grounding, prediction feedback, feature-level SHAP feedback, and iterative refinement.

6. Reproducibility: The paper details complete prompt templates, indexing of retrieved evidence, and candidate code generation contracts. RF training and evaluation protocols are fixed and documented. The data and code release status is unclear, but extensive archived run artifacts and detailed supplementary material enable partial reproducibility. However, the clinical image data is restricted due to privacy.

Concrete example: At round t=1, the LLM uses a program generation prompt describing the KD vs HS classification task and clinical scar scales (e.g., Vancouver Scar Scale). It issues queries to a web search operator that returns passages from peer-reviewed literature describing visual attributes such as pigmentation and vascularity. Based on retrieved evidence, the LLM generates M=4 Python programs computing features like scar area, color histograms, and texture statistics, each dimension annotated with name, clinical definition, and source citation. These programs execute locally on Dfit images to produce feature matrices. An RF is trained on these features, validated on Dval, and metrics like balanced accuracy and SHAP values are computed. The LLM receives only the aggregate feedback (no images or per-patient outputs) and uses them to repair code, remove unsupported features, or add new clinically relevant features at round t=2. The process repeats for T=3 rounds before final program selection and retraining on full development data.

Technical innovations

• Translate LLM clinical knowledge into deterministic, locally executable, traceable feature programs instead of end-to-end or direct VLM image classification.
• Validation-guided iterative program synthesis loop uses aggregate metrics and SHAP feature importances to repair and refine code without direct image or patient-level data sharing.
• Separation of roles—LLM for clinical knowledge retrieval and program generation, local sandboxed feature extraction, and classical machine learning classification—enabling auditability and privacy compliance.
• Use of SHAP feature attributions in program synthesis feedback to identify useful vs spurious features, promoting semantically aligned refinement.
• Demonstration of cross-site generalization via leave-one-site-out evaluation and robustness under severe data scarcity (only 10% development data).

Datasets

• Scar pathology dataset — 600 clinical photographs from 3 hospitals (200 images per hospital) — private multi-institutional collection with expert-labeled keloid and hypertrophic scars

Baselines vs proposed

• Handcrafted+RF: site-macro balanced accuracy = 61.3% vs ScaFE: 81.0%
• ResNet-18 fine-tuned: 64.2% vs ScaFE: 81.0%
• EfficientNet-B0 fine-tuned: 65.2% vs ScaFE: 81.0%
• DINOv3 + linear probe: 68.7% vs ScaFE: 81.0%
• Derm Foundation + probe: 69.7% vs ScaFE: 81.0%
• BiomedCLIP + linear probe: 71.0% vs ScaFE: 81.0%
• Local VLM-direct inference: 66.2% vs ScaFE: 81.0%
• Random ResNet features: ~51.2% (chance level) vs ScaFE: 81.0%
• Label-permutation control: ~50.0% vs ScaFE: 81.0%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.28538.

Fig 1: Overview of ScaFE. An LLM turns source-traceable clinical concepts into deterministic feature programs that process

Fig 2 (page 4).

Fig 3 (page 4).

Fig 4 (page 4).

Limitations

• Retrospective two-class classification study lacks prospective clinical validation.
• Focuses only on visual scar attributes; nonvisual clinical data (e.g. palpation, patient history) omitted.
• Restricted to three hospitals; generalization to other sites or populations untested.
• LLM-generated program synthesis requires fixed downstream learner and controlled iterative refinement; applicability to other tasks unclear.
• Potential computational cost of iterative LLM queries and web retrieval may limit scalability.
• No explicit adversarial evaluation for manipulated input images or feature programs.

Open questions / follow-ons

• Can the ScaFE approach generalize beyond scar classification to other medical imaging tasks with complex clinical knowledge?
• How does the choice of downstream learner affect generalization and interpretability across feature-program variants?
• What are the effects of different LLM architectures, evidence sources, or synthesis budgets on program quality and efficiency?
• How resilent is the approach against adversarial inputs or attempts to subvert evidence-grounded feature extraction?

Why it matters for bot defense

From a bot-defense and CAPTCHA perspective, ScaFE exemplifies a novel paradigm whereby large language models contribute clinical domain knowledge not by direct prediction on sensitive data, but by producing interpretable, auditable feature extraction functions that run locally. This approach mitigates privacy risks associated with cloud-hosted multimodal inference and improves reproducibility by decoupling knowledge transfer from patient data access. In bot defense, a comparable principle could guide designing interpretable feature programs generated or refined by LLMs under strict audit and validation, rather than relying on opaque end-to-end models. The iterative feedback loop using aggregate metrics and feature importance resembles adversarial testing and robustness improvement cycles familiar in CAPTCHA design, suggesting a potential methodological synergy. However, ScaFE’s reliance on costly LLM queries and clinical evidence retrieval would require adaptation for large-scale, low-latency CAPTCHA environments.

Cite

@article{arxiv2607_28538,
  title={ ScaFE: Data-Efficient Scar Classification with LLM-Generated Clinical Feature Programs },
  author={ Ruman Wang and Hangting Ye },
  journal={arXiv preprint arXiv:2607.28538},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.28538}
}

Read the full paper

• arXiv:2607.28538 (abstract)
• arXiv:2607.28538 (PDF)