Validity of LLMs as data annotators: AMALIA on authority
Source: arXiv:2607.08731 · Published 2026-07-09 · By Manuel Pita
TL;DR
This paper investigates the validity of large language models (LLMs) as instruments for annotating theoretical constructs in text, focusing on AMALIA-9B, Portugal's national 9-billion-parameter model specialized in European Portuguese. While AMALIA achieves agreement levels within six F1 points of much larger open models on coding the moral foundation of authority, agreement alone does not establish construct validity. Using a decomposition technique called grain calibration, the study probes whether AMALIA measures the authority construct according to its underlying theory or relies on correlated shortcuts such as moral outrage near authority figures. The study finds a substantial recovery gap: decomposing AMALIA's holistic prompt into the construct's atomic clauses recovers only about half of its original performance, indicating that much of AMALIA's authority coding arises from surface-level correlates rather than the formal theoretical inference. In contrast, an open multilingual LLM closes this recovery gap on the same Portuguese corpus, suggesting the problem lies in the model and prompt rather than the corpus itself. The paper argues that national LLM programs must evaluate not just annotator agreement but also validity diagnostics to ensure models genuinely measure theoretical constructs rather than proxy patterns. AMALIA may still be useful for large-scale screening and pre-coding but is not yet a standalone instrument for measuring complex constructs like authority.
Key findings
- AMALIA-9B agrees with human coders on authority annotation within six F1 points of open models 8-13x its size.
- Decomposition recovers only about 50% of AMALIA's holistic prompt performance on authority construct annotation (a large recovery gap).
- An open multilingual LLM closes the recovery gap on the same Portuguese corpus and instructions, indicating corpus is not the limiting factor.
- AMALIA's authority annotations often rely on surface correlates such as moral outrage near authority figures rather than the theory's inference steps.
- Use of English vs Portuguese prompt instructions for AMALIA does not close the recovery gap or improve construct validity significantly.
- The confirmatory test used 448 unseen texts from a 748-text European Portuguese transcreation of the Moral Foundations Reddit Corpus.
- Parse rates for AMALIA annotations were nearly 100%, showing high reliability in producing output formats.
- The recovery gap ∆ ≥ 0.10 was observed for both English and Portuguese prompts with 95% confidence.
- False-positive errors were mostly attributable to shortcut bases, with more than half rated as partially grounded or less by an independent LLM audit panel.
Threat model
n/a — the paper does not target a hostile adversary but rather evaluates whether an LLM annotation instrument legitimately measures a theoretical construct versus obtaining correct labels through shortcuts or correlations.
Methodology — deep read
The study assesses AMALIA-9B's validity as an annotation instrument for the moral foundation construct authority/subversion, originally defined in English and here transcreated into European Portuguese. The threat model assumes the adversary is absence: the work aims to confirm that AMALIA produces construct-valid codes, not that it resists adversarial attacks. The dataset is a European Portuguese transcreation (Cpt) of 748 texts from the Moral Foundations Reddit Corpus (MFRC), preserving referents, stance, and illocutionary force, performed by multilingual LLMs and involving human adjudication for quality.
The annotation instrument is a prompt operationalizing the theory-based codebook for authority: a nine-clause prompt with a decision rule (D ∧ (A1 ∨ A2)) ∨ A3 ⇒ authority, where D is detection of authority-related anchors, and A1, A2, A3 represent appraisal clauses. There is an undecomposed (holistic) prompt that asks the model to annotate authority in one step and a decomposed prompt that asks the model to answer each clause independently; clause results are combined deterministically.
Three LLMs are tested: AMALIA-9B under Portuguese (P_pt) and English (P_en) prompts, and two large open multilingual models (Llama-3.3-70B and GPT-OSS-120B) under English prompts only. The primary endpoint is the recovery gap ∆ = F1_undecomposed - F1_decomposed on the positive authority class against gold human labels; it quantifies whether decomposing the prompt faithfully reproduces the holistic prompt performance.
Model outputs were generated at zero temperature with deterministic JSON-constrained output; fallback tiers handled parsing failures. The recovery gap was assessed on a pre-registered set of 448 held-out texts unseen during model prompting. Interpretative thresholds for ∆ were pre-registered: open gap if ∆ ≥ 0.10, closed if ∆ < 0.05.
Additional analyses include error analyses with independent LLM readers under charitable and skeptical viewing perspectives, rating false positives for grounding and shortcut evidence. Parsing rates and robustness runs without output constraints were performed. Cross-validation was not explicitly mentioned; reproducibility is supported by deposited protocols and frozen corpora, but AMALIA's weights are closed.
One concrete example: for a given text, the undecomposed prompt outputs an authority/no answer in one call. The decomposed prompt queries each clause separately; results are recombined. If the recombination matches the holistic call, the recovery gap is zero. For AMALIA, half of its agreement performance could not be reconstructed from clause-level answers, evidencing reliance on correlations beyond the construct's theory.
Technical innovations
- Application of grain calibration decomposition to isolate clause-level prompts aligning with theoretical construct definitions for LLM annotation validity assessment.
- Introduction of the recovery gap metric ∆ to quantify validity by measuring performance loss when decomposing a holistic prompt into theory-defined atomic clauses.
- First validity study of a national language model (AMALIA-9B) for theoretical construct annotation beyond agreement metrics, testing transfer of a calibrated English prompt instrument to European Portuguese.
- Use of LLM audit panels to systematically evaluate grounding and shortcut biases in false-positive model errors on complex construct annotation.
Datasets
- Moral Foundations Reddit Corpus (MFRC) — 748 texts (after deduplication and transcreation to European Portuguese) — transcreated version not publicly released
- Original English MFRC — original corpus for ground truth annotation
Baselines vs proposed
- AMALIA-9B undecomposed prompt F1 on authority: around 0.6 (exact number not stated) with Portuguese prompt
- AMALIA-9B decomposed prompt F1 on authority: about half of undecomposed (∼0.3), recovery gap ∆ ≥ 0.10 indicating large validity gap
- GPT-OSS-120B undecomposed prompt F1 on authority: higher than AMALIA (exact value not stated), recovery gap ∆ ≈ 0 meaning nearly complete recovery
- Llama-3.3-70B recovery gap around 0.05 on English corpus, near threshold of tolerance
- Parse rate: AMALIA 100%, GPT-OSS 98.3%, Llama 100% parsing with JSON constraints
Limitations
- Study focuses on a single theoretical construct (authority) and one corpus, limiting generalization to other constructs or datasets.
- AMALIA was tested with a transferred English-calibrated prompt, not with a prompt specifically calibrated for AMALIA or Portuguese.
- Residual risk of semantic drift during transcreation of English texts to European Portuguese despite verification and human adjudication.
- No adversarial evaluation or test under domain/temporal distribution shifts was performed.
- AMALIA’s closed pretrained weights and training data limit reproducibility and independent verification.
- Error analysis relies partly on subjective LLM auditing panels without independent human coder verification.
Open questions / follow-ons
- Can prompt calibration tailored specifically for AMALIA or Portuguese close the recovery gap and improve construct validity?
- Does incorporating native-language instruction and localized data reduce shortcut reliance in sovereign LLMs generally?
- How do other theoretical constructs and annotation tasks behave with this grain calibration decomposition approach on national models?
- Can an open multilingual LLM outperform national models consistently in validity and scalability for text-as-data annotation?
Why it matters for bot defense
This paper highlights a critical distinction between agreement (reliability) and validity when using LLMs as annotators for complex theoretical constructs in text—a distinction directly relevant for bot-defense engineers relying on labeled data or behavioral signals. While a national language model like AMALIA may agree substantially with human labels, it can still arrive at those labels through proxy cues unaligned with the theoretical intent, risking invisible systematic errors. For CAPTCHA and bot-defense tasks that depend on nuanced semantic judgments or behavioral intents, similar validity checks via decomposition or interpretability analyses should complement simple agreement metrics. Moreover, the finding that larger multilingual LLMs can better align with theoretic constructs suggests that smaller specialized sovereign models may require additional calibration or supervision before being deployed as automated annotation instruments at scale. Overall, the recovery gap metric and grain calibration methodology represent promising tools for assessing and improving the validity of LLM annotations in security-sensitive text labeling, including CAPTCHA datasets or user intent classification.
Cite
@article{arxiv2607_08731,
title={ Validity of LLMs as data annotators: AMALIA on authority },
author={ Manuel Pita },
journal={arXiv preprint arXiv:2607.08731},
year={ 2026 },
url={https://arxiv.org/abs/2607.08731}
}