Aristotelian Virtue Profiling of LLMs through Ethical Dilemmas

Source: arXiv:2606.28683 · Published 2026-06-27 · By Ioannis Tzachristas, John Pavlopoulos

TL;DR

This paper addresses the challenge of evaluating the ethical decision-making behavior of large language models (LLMs) beyond binary right-or-wrong judgments. The authors introduce VirtueMap, a novel framework grounded in Aristotelian virtue ethics that profiles model responses to ethical dilemmas across five virtues: Practical Wisdom, Justice, Truthfulness, Courage, and Temperance. Instead of selecting a single "correct" answer, VirtueMap asks human raters and LLMs to rank all five responses to each of seven non-lethal, non-political dilemmas. The rankings are scored against validated virtue-expression orderings using a normalized Borda alignment metric, producing continuous virtue profiles. Applying VirtueMap to nine LLM families revealed consistently high Practical Wisdom scores but more variation in Courage, Temperance, and Justice, indicating meaningful differences in how models handle ethical tradeoffs. The authors also provide an interactive website enabling local, private profiling and comparison of humans and LLMs using this framework.

VirtueMap's key innovation is its descriptive, rather than prescriptive, approach to ethical evaluation. By anchoring on Aristotelian virtue ethics and extensively validating virtue-expression orderings with over 100 human raters per ordering (retaining only those with ≥95% agreement), it moves beyond typical moral benchmarks that seek a single "correct" choice. The framework yields interpretable five-dimensional virtue profiles capturing nuanced ethical preferences and tradeoffs expressed by models and humans. This multidimensional, ranking-based evaluation facilitates a richer understanding of ethical behaviors and distinctions between models in contexts that admit multiple defensible responses.

Key findings

• The mean virtue-profile score across nine LLM families is highest for Practical Wisdom at 90.4 and lowest for Temperance at 76.9, with Truthfulness (82.3), Justice (80.5), and Courage (78.0) in between.
• The largest cross-model variation appears on Courage (10.95 point range), Temperance (10.26), and Justice (9.70), highlighting these virtues as discriminative dimensions.
• Mean rank consistency of repeated full-questionnaire runs per model is 90.3% (converted from Kendall's τ), indicating stable LLM rankings under stochastic sampling.
• Virtue-expression orderings were retained only if >100 raters confirmed them with ≥95% agreement, ensuring the operational ground truth derives from strong common-sense consensus.
• LLMs differ in virtue strengths: for example, Claude scores highest on Truthfulness (86.0) and Courage (83.1), Qwen on Practical Wisdom (93.4), MiniMax on Justice (85.4), and GPT on Temperance (82.3).
• Random permutation of response options per run and JSON output enforcement helped ensure robust, unbiased ranking collection.
• The interactive website runs all computations locally in-browser without storing user responses, supporting privacy-preserving human-LLM ethical profile comparisons.
• The normalized Borda alignment metric provides a quantitative score on a 0-100 scale reflecting alignment with validated virtue orderings per dilemma and virtue.

Threat model

n/a — This work is not a security paper and does not define an adversary or attacker capabilities. It is an ethical evaluation framework for profiling and comparing LLM behavioral patterns under ethical dilemmas, without security or adversarial threat considerations.

Methodology — deep read

1. Threat Model & Assumptions: The study assumes an ethical evaluator role rather than an adversarial security threat model. The "adversary" in this context is omitted, as the focus is on characterizing LLM ethical behavior patterns, not attacking or evading defenses.

2. Data: The framework uses seven general ethical dilemmas (Spreadsheet Error, Deadline Exception, Early Warning, Taking Responsibility, Favor Request, Public Explanation, Allocation Decision) designed to be non-lethal, non-religious, and apolitical. Each dilemma has 5 fixed response options (A-E). For each dilemma and each of 5 virtues, the authors propose an ordering of the 5 responses ranked from most to least expressive of that virtue. These proposed virtue-expression orderings were then validated via online surveys with >100 respondents per ordering. Only orderings receiving at least 95% confirmation from respondents were retained as "operational ground truth."

3. Scoring Architecture: Human or LLM rankings of responses per dilemma (complete strict orderings of all five options) are scored against the retained virtue-expression orderings using normalized Borda alignment. The Borda scoring assigns descending point weights (5,4,3,2,1) based on rank positions. The alignment score is the weighted rank correlation between observed and virtue orderings, normalized linearly so that perfect alignment maps to 100 and complete reversal to 0. Averaging over dilemmas yields a 0-100 virtue profile score for each virtue.

4. Training/Prompting LLMs: Nine LLM families (GPT, Claude, Gemini, Llama, DeepSeek, Mistral, MiniMax, Grok, Qwen) were evaluated via the OpenRouter API. For each model and dilemma, the five responses were randomly permuted before prompting. Models were then instructed to return a complete, tie-free ranking of the permuted options in JSON format. Invalid outputs were discarded and re-queried. Each model underwent 3 to 10 valid full-questionnaire runs to measure stability.

5. Evaluation Protocol: The authors computed mean and 95% confidence intervals of virtue scores across runs per model. Pairwise Kendall’s tau between repeated runs was computed to assess rank consistency per model, rescaled to a 0-100 stability percentage. The run suite stopped collecting more runs once confidence intervals were sufficiently tight or consistency exceeded thresholds. This sequential protocol treats LLM output stochasticity as an intrinsic characteristic rather than noise.

6. Reproducibility: The authors released code, questionnaire data, evaluation scripts, and an interactive website that replicates the methodology, enabling local scoring of human or model rankings without central data collection. The prompt templates, dilemmas, and validated ground-truth virtue orderings are documented fully in appendices. However, exact model versions used and decoding parameters are not exhaustively detailed. The datasets and code are public, supporting replicability.

Concrete example end-to-end: For the Spreadsheet Error dilemma, five candidate responses (A-E) are presented. The validated ground-truth ordering for Courage might be A > B > C > D > E. An LLM run will randomly permute the order of response options, prompt the model to rank all five with no ties in JSON, then map the returned ranking to the canonical labels. The normalized Borda alignment score S(d,v) compares this ranking to the ground-truth Courage ordering. This process repeats for all seven dilemmas and five virtues, and the per-virtue scores are averaged into the final profile. Multiple runs per model allow estimating rank consistency and confidence intervals.

Technical innovations

• A ranking-based ethical evaluation framework for LLMs grounded explicitly in Aristotelian virtue ethics, profiling five virtues instead of seeking single correct answers.
• Common-sense validation of virtue-expression orderings with large human rater panels using a strict ≥95% confirmation threshold to operationalize ground truth.
• Use of normalized Borda alignment to quantitatively score how closely rankings align with virtue priorities, producing continuous, interpretable virtue profiles.
• A repeated-run LLM evaluation protocol that treats model output stochasticity as signal rather than noise, computing stability via Kendall’s tau.
• An interactive, privacy-preserving web application enabling local computation and comparison of human and LLM ethical virtue profiles.

Datasets

• VirtueMap Ethical Dilemma Dataset — 7 dilemmas × 5 responses each — constructed by authors, publicly documented in appendix and website
• Human validation dataset — >100 responses per dilemma-virtue ordering pair — collected online via questionnaire, confirmed ≥95% for retained orderings

Baselines vs proposed

• No explicit baseline models reported; comparison is across nine LLM families under identical evaluation.
• Rank consistency (Kendall’s τ scaled to 0–100): Claude 96.2 vs GPT 91.2 vs Llama 87.3; average 90.3.
• Virtue scores (0–100 scale): Practical Wisdom GPT 92.3 vs Claude 89.3 vs Qwen 93.4 vs MiniMax 87.9.
• Truthfulness Claude 86.0 vs GPT 81.1 vs Gemini 82.9; Courage Claude 83.1 vs MiniMax 83.0 vs GPT 76.3; Temperance GPT 82.3 vs Qwen 80.4 vs Claude 72.6.

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.28683.

Fig 1: VirtueMap workflow. Dilemmas and response rankings are grounded through common-sense

Fig 2: Pentagon profiles for the nine measured LLM families. Scores are computed on 0–100 and

Fig 3 (page 9).

Fig 4 (page 9).

Fig 5 (page 9).

Fig 6 (page 10).

Fig 7 (page 10).

Fig 8 (page 10).

Limitations

• The dataset consists of only seven dilemmas and five virtues, limiting the breadth of ethical scenarios and traits profiled.
• Virtue-expression orderings are validated through a confirmation/correction task rather than fully blind rankings, possibly inflating agreement due to confirmation bias.
• Dilemmas exclude lethal, religious, and political content; results may not generalize to more charged or complex ethical issues.
• Models' virtue profiles may depend on model versions, decoding settings, prompt formats, API routing, and number of runs, limiting direct generalizability.
• The framework captures preference patterns but does not imply LLMs truly possess moral character or reason morally.
• Respondents and LLMs may make different assumptions about missing context in dilemmas, influencing rankings.

Open questions / follow-ons

• How stable are virtue profiles across broader, more diverse sets of dilemmas, including lethal, political, or religious scenarios?
• Can validation of virtue-expression orderings be improved by fully blind ranking tasks rather than confirmation/correction questionnaires?
• How do LLM prompt engineering, decoding methods, and model versions affect virtue profiles and their interpretability?
• Could additional virtues or ethical frameworks be incorporated to enrich and diversify profile dimensions?

Why it matters for bot defense

Bot-defense and CAPTCHA practitioners concerned with nuanced AI behavior evaluation can use VirtueMap as a methodological example of multidimensional ethical profiling rather than binary correctness testing. The virtue-based ranking approach captures subtleties in model preferences and tradeoffs, providing richer signals for interpretable behavioral distinctions. This could aid in designing behavioral challenges or response assessments that probe models' ethical dispositions or reluctance to produce certain content types.

Furthermore, VirtueMap’s repeated-run protocol and stability assessments offer a model evaluation paradigm that integrates output randomness as a meaningful factor, potentially inspiring more robust bot-detection features based on behavioral consistency measures. However, the framework currently targets ethical dilemma reasoning rather than typical CAPTCHA interactions, so adaptation would be required for direct application in bot defense scenarios.

Cite

@article{arxiv2606_28683,
  title={ Aristotelian Virtue Profiling of LLMs through Ethical Dilemmas },
  author={ Ioannis Tzachristas and John Pavlopoulos },
  journal={arXiv preprint arXiv:2606.28683},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.28683}
}

Read the full paper

• arXiv:2606.28683 (abstract)
• arXiv:2606.28683 (PDF)