Measuring and Evaluating the Performance of Generative AI Models for Scam Detection

Source: arXiv:2607.17353 · Published 2026-07-19 · By Cem Topcuoglu, Seyed Ali Akhavani, Harel Berger, Sadia Afroz, Michalis Pachilakis, Vibhor Sehgal et al.

TL;DR

This study rigorously evaluates the effectiveness of large language models (LLMs) for detecting online scams across multiple real-world scenarios without task-specific fine-tuning. The authors introduce ScamBenchmark, a unique multi-source dataset of 2,742 labeled scam, clean, and uncertain samples from Reddit, Google Images, Mechanical Turk, and proprietary data totaling over 59,000 samples. They evaluate nine LLMs including Mistral, Llama 3 variants, and OpenAI’s ChatGPT models employing various prompting strategies (zero-shot, few-shot, chain-of-thought, common sign injection, and URL intelligence) and compare their performance against a fine-tuned BERT classifier. The results show that larger LLMs generally achieve higher detection accuracy, with ChatGPT 4o1 and Llama 3.1 70B leading at micro-F1 scores near 64-65%. Thoughtful prompt engineering boosts smaller models' performance substantially, although prompting effects vary by model. LLMs also demonstrate stronger generalization to unseen scams compared to the fine-tuned BERT baseline, highlighting the value of pre-trained knowledge in combating novel threats. The authors open source their dataset and evaluation framework to foster more robust scam detection research.

Key findings

• ChatGPT 4o1 achieved the highest micro-F1 score of 64.9% on scam detection using zero-shot main prompts, outperforming other models.
• Llama 3.1 70B closely followed with a micro-F1 around 61-63%, outperforming smaller open-source and proprietary LLMs.
• Fine-tuned BERT classifiers scored significantly lower, with weaker generalization to unseen scams compared to LLMs.
• Prompting method and model interact: e.g., chain-of-thought prompting improved performance for smaller models but degraded it for some larger ones.
• Injecting common scam sign lists into prompts improved micro and macro F1 scores for most LLMs except ChatGPT 4o and 4o1.
• URL intelligence integration had mixed results, sometimes reducing performance compared to other prompt types.
• Increasing shots in few-shot prompting generally enhanced performance, with 10-shot prompting raising Llama 3.1 70B’s micro F1 to 65.3%, surpassing ChatGPT 4o1 zero-shot.
• Smaller models (e.g., Mistral 7B) benefited most from careful prompting but still trailed larger LLMs by 10-15 percentage points in F1.

Threat model

The adversary is an online scammer deploying diverse types of scams across textual communication channels aiming to evade automated detection. The adversary crafts scam messages possibly including URLs but cannot directly influence model prompts or training data. The detection system is assumed to receive text inputs and optional URL analysis but has no access to private or contextual metadata. The study evaluates model performance under zero/few-shot prompting without attacker adaptation or adversarial manipulation of model inputs.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is an online scammer attempting to evade detection by generating scam content including phishing, fraudulent offers, and social engineering messages. The models do not have access to any additional context beyond the text input and optional external URL intelligence. Models are evaluated on their ability to classify samples as Scam, Clean, or Uncertain. No adversarially crafted inputs were tested.

2. Data: The authors curated a labeled dataset of 2,742 samples from Reddit's 'Is this a scam?' flair (1,270 samples), Google Images (615 samples), and Mechanical Turk (857 samples), with labels Scam, Clean, or Uncertain. Additionally, a large proprietary dataset with 59,991 samples was used for validation. The dataset spans 25 scam categories, emphasizing fine-grained labeling. Text was obtained from posts, messages, and OCR of images, with PII redacted. Human annotators labeled samples; uncertain labels reflected ambiguous cases.

3. Architecture/Algorithm: Nine LLMs ranging from 7B up to 70B parameters were evaluated: Mistral 7B and 8x7B, Llama 3 8B, Llama 3.1 (8B, 70B), Llama 3.2 11B, and proprietary ChatGPT models (4, 4o, 4o1). Models were prompted using five strategies: zero-shot main prompt (direct classification), chain-of-thought reasoning prompt (generate arguments for/against scam before classifying), common signs prompt (inject known scam characteristics), URL intelligence prompt (inject URL trust scores), and few-shot prompting (providing labeled examples). Additionally, Mistral 7B, Llama 3.1 8B, and a BERT model were fine-tuned on balanced subsets.

4. Training Regime: Fine-tuning was conducted on small datasets with 300-1,200 examples across classes, using epoch ranges 1-4, batch sizes 16-32, and learning rates 1e-5 to 5e-5. Experiments on open source LLMs used two NVIDIA A100 GPUs. For proprietary models, Generative AI APIs with black-box access were used. Each experiment was run 5 times with majority voting.

5. Evaluation Protocol: Primary metrics were micro and macro-averaged precision, recall, and F1 scores across the three classes. Specific class-wise metrics were also reported. Statistical significance testing was not detailed. Models were tested on held-out dataset splits with unseen scams to evaluate generalization. Ablations on prompt types, number of shots, and temperature/top-p hyperparameters were carried out.

6. Reproducibility: The authors open-sourced the ScamBenchmark dataset and evaluation framework on GitHub. Fine-tuned model weights for some open-source models were also released. Proprietary model weights remain closed. Prompt templates and hyperparameter settings were provided in detail. Code to replicate experiments and run evaluations was included.

Concrete Example End-to-End: For zero-shot main prompt evaluation, a given input text (e.g., “Confirm your bank details to keep your account active”) is prepended with the instruction "Classify the following input into Scam, Clean or Uncertain" and fed to the LLM. The model outputs a classification token suitable for automated parsing. Results are compared against ground truth for metrics like micro F1. Repeating 5 times mitigates variance. This process is repeated with different prompting strategies, models, and hyperparameters to assess performance comprehensively.

Technical innovations

• Introduction of ScamBenchmark, the first multi-source, fine-grained labeled dataset specifically designed to evaluate LLMs for scam detection including uncertain labels.
• Systematic evaluation of nine diverse LLMs with varied parameter sizes and architectures on scam detection using multiple prompting strategies (zero-shot, chain-of-thought, few-shot, and domain knowledge injection).
• Demonstration that carefully engineered prompts and few-shot examples can substantially enhance smaller LLMs’ scam detection performance, narrowing the gap to larger models.
• Novel integration of external URL safety intelligence (from Tranco and ScamAdviser) into LLM prompts to augment scam classification.
• Empirical comparison highlighting that LLMs generalize better to unseen, diverse scam samples than traditional fine-tuned classifiers like BERT.

Datasets

• ScamBenchmark — 2,742 samples — public (Reddit, Google Images, MTurk)
• Proprietary Scam Dataset — 59,991 samples — industry partner (private)

Baselines vs proposed

• Fine-tuned BERT: micro-F1 ≈ 0.45 vs ChatGPT 4o1 Zero-shot main prompt: micro-F1 = 0.649
• Fine-tuned Mistral 7B: micro-F1 ≈ 0.48 vs Mistral 8x7B Zero-shot main prompt: micro-F1 = 0.507
• Llama 3.1 70B Zero-shot main prompt: micro-F1 = 0.610 vs ChatGPT 4o1 Zero-shot main prompt: micro-F1 = 0.649
• Llama 3.1 70B 10-shot prompt: micro-F1 = 0.653 vs ChatGPT 4o1 Zero-shot main prompt: micro-F1 = 0.649

Limitations

• The evaluation does not test robustness against adversarially crafted scam inputs designed to evade detection.
• Proprietary dataset remains closed, limiting full reproducibility and external validation beyond ScamBenchmark.
• Prompt engineering effectiveness varies widely by model, complicating universal deployment recommendations.
• Hyperparameter tuning is limited to small temperature and top-p sets; larger parameter search may yield better configs.
• LLMs evaluated solely on text inputs without multi-modal signals (images, behavioral data), which could improve real-world detection.
• Uncertain class labeling remains subjective and may impact performance metrics; no external metadata was used in evaluation.

Open questions / follow-ons

• How do LLMs perform on adversarially crafted scam messages explicitly designed to bypass detection models?
• Can multi-modal approaches combining text with images, metadata, and behavioral signals improve scam detection over text-only LLM methods?
• What is the impact of temporal concept drift in scams on LLM scam detection generalization over time?
• How can prompt engineering be further automated or personalized per model to consistently maximize scam detection accuracy?

Why it matters for bot defense

For bot-defense practitioners and CAPTCHA designers, this paper provides critical insights into leveraging large language models for scam detection, a related domain where discerning malicious intent in user-generated content is paramount. The findings underscore that larger LLMs, paired with tailored prompt engineering, can effectively identify scams better than traditional classifiers and generalize to unseen scam types, suggesting a promising role for generative AI in identifying fraudulent bot activity or social engineering attempts. However, the variable impact of different prompts and temperature settings implies that integration into production CAPTCHA or bot detection systems requires careful model- and context-specific tuning. Furthermore, the demonstrated benefit of injecting domain knowledge and auxiliary URL intelligence hints at hybrid models combining LLM reasoning with external threat intelligence could enhance overall bot-defense robustness. The release of the ScamBenchmark dataset also offers a valuable benchmarking resource for evaluating language-based fraud detection systems in authentication flows.

Cite

@article{arxiv2607_17353,
  title={ Measuring and Evaluating the Performance of Generative AI Models for Scam Detection },
  author={ Cem Topcuoglu and Seyed Ali Akhavani and Harel Berger and Sadia Afroz and Michalis Pachilakis and Vibhor Sehgal and Leyla Bilge and Engin Kirda },
  journal={arXiv preprint arXiv:2607.17353},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.17353}
}

Read the full paper

• arXiv:2607.17353 (abstract)
• arXiv:2607.17353 (PDF)