Why Machines Misread Pedagogical Quality: Human-Machine Alignment in LLM-Based Pretest Question Evaluation
Source: arXiv:2606.23629 · Published 2026-06-22 · By Pei-Yu Tseng, Mahir Akgun, Peng Liu
TL;DR
This paper addresses the challenge of scaling high-quality pretest question generation and evaluation using large language models (LLMs). Designing pedagogically sound pretest questions involves considerations of openness, cognitive depth, relevance, and clarity, which traditionally require expert manual effort. The authors propose an AI-assisted workflow combining automated question generation grounded in instructional materials, rubric-based evaluation, and iterative selection to produce high-quality pretests at scale. The core focus is on investigating the alignment between human raters and machine evaluators when assessing question quality via rubric scoring. They conduct a 2×2 factorial study varying rubric operationalization (initial human-defined rubric vs. a revised rubric explicitly adapted for machine use) and evaluation mode (direct scoring vs. rationale-first, where the model explains reasoning before scoring). Results demonstrate that human-machine disagreements are systematic rather than random and mainly arise from ambiguous rubric definitions, particularly for the openness and depth dimensions. Revising the rubric to provide operationalized, explicit criteria improved alignment substantially more than rationale-first prompting alone. However, the highest alignment was achieved by combining rubric revision with rationale-first evaluation, underscoring the complementarity of clear construct definitions and transparent model reasoning. This work highlights that scalable AI-assisted pedagogical evaluation depends critically on how rubric constructs are formalized for machine interpretation, beyond generation capability alone.
Key findings
- Under the initial rubric with direct evaluation, openness agreement between human and machine was 55.7%, with a positive bias of 0.66 and MSE of 1.08, indicating systematic underestimation of openness by the machine.
- Improving rubric operationalization (revised rubric) increased openness agreement to 83.3%, reduced bias to 0.22, and lowered MSE to 0.32, while depth agreement improved from 70.5% to 95.0%.
- Rationale-first evaluation alone (initial rubric) increased depth agreement from 70.5% to 87.0% but only marginally improved openness agreement (55.7% to 63.0%).
- Combining revised rubric with rationale-first evaluation achieved the highest alignment: openness agreement of 94.5% (MAE and MSE 0.05) and depth agreement maintaining 95.0%.
- Relevance and clarity dimensions showed near-perfect human-machine agreement (up to 100%), as they involve more surface-tractable features.
- The systematic disagreement patterns stemmed mostly from rubric operationalization ambiguities rather than LLM inconsistency.
- Rubric revision employed principles such as separating response-space breadth from surface cues and operationalizing scoring with explicit decision rules and examples.
- Iterative generation and filtering over 15 rounds produced approximately 120 high-quality candidate questions per rubric version, covering a balanced 3x3 openness–depth space.
Threat model
The adversary is not an explicit attacker but the challenge is that automated machine evaluation of pretest question quality may systematically misalign with human expert judgments due to ambiguous rubric definitions or superficial model heuristics. The machine evaluator is assumed to have access to the rubric, learning materials, context, and generated questions but cannot access human rationale beyond the rubric. The risk is over- or underestimating pedagogical quality in automated filtering leading to suboptimal question selection.
Methodology — deep read
The study sets out to improve scaling of pretest question generation and automated evaluation using LLMs, focusing on human-machine alignment in rubric-based scoring of question quality. The threat model includes reliance on machine-generated evaluations to filter questions at scale, assuming human expert ratings as ground truth.
Data provenance involves questions generated by GPT-5.1 within a cybersecurity statistics course context. Each of two rubric versions (initial human-defined and a revised operationalized one) guided separate generation runs of 15 rounds with 12 candidate questions each (180 total). Post filtering for relevance and clarity below level 3, about 120 questions per rubric version remained. Human expert raters scored samples of 60 questions per rubric to provide reference labels.
The rubric evaluates four dimensions: openness, depth (cognitive demand per Bloom's taxonomy), relevance, and clarity, each on 3-level ordinal scales. The revised rubric explicitly clarifies scoring criteria with classification structures, examples, and decision rules to reduce ambiguity.
The LLM evaluator assigns rubric scores under two modes: direct evaluation (immediate scoring) and rationale-first evaluation (articulating rubric-grounded reasoning before scoring), enabling a 2x2 factorial design across rubric and evaluation mode.
Human-machine alignment metrics include bias (directional human minus machine score differences), mean absolute error (MAE), mean squared error (MSE), and exact agreement percentage.
The generation–evaluation workflow iteratively generates question batches, scores and filters them based on rubric scores and pairwise comparisons, then analyzes coverage gaps in the openness–depth space to guide subsequent generation rounds. This design supports diversity and balanced coverage.
The experimental procedure: generate and evaluate 180 questions per rubric version, filter to ~120 questions, human score subsets, and compare against machine scores in four evaluation conditions. Human inter-rater agreement improved from Cohen's κ=0.73 under initial rubric to 0.78 with revised rubric, confirming better clarity.
Key evaluation outcomes are reported in Table 2, with detailed analysis of discrepancy patterns prompting rubric revisions. Rationale-first evaluation prompts the model to explicate intermediate reasoning in rubric terms, supporting more principled scoring calibration.
Code and data availability are not explicitly mentioned. The dataset is proprietary to this study and not public. The paper does not report external cross-validation but evaluates alignment across multiple conditions and independent human raters to ensure robustness.
Technical innovations
- Operationalizing pedagogical rubric constructs with explicit decision rules and classification structures tailored for LLM evaluation, improving interpretability and scoring consistency.
- Combining rubric revision with rationale-first evaluation prompting to jointly improve human-machine alignment in question quality assessment.
- A multi-round iterative generation and filtering workflow balancing question diversity across a 3 × 3 openness–depth space using paired comparisons and quality thresholds.
- Systematic analysis of human-machine scoring disagreement patterns to derive rubric design principles for representational clarity and construct validity.
Datasets
- Generated Pretest Questions in Cybersecurity Statistics — ~240 questions total (120 per rubric version) — internal, non-public
Baselines vs proposed
- Initial rubric + direct evaluation: openness agreement = 55.7% vs revised rubric + direct evaluation: 83.3%
- Initial rubric + direct evaluation: depth agreement = 70.5% vs revised rubric + direct evaluation: 95.0%
- Initial rubric + direct evaluation: openness bias = 0.66 vs revised rubric + direct evaluation: 0.22
- Initial rubric + rationale-first evaluation: depth agreement = 87.0% vs revised rubric + rationale-first evaluation: 95.0%
- Relevance and clarity dimensions: perfect or near-perfect agreement (~100%) across all conditions
Limitations
- Focus is on alignment between human and machine rubric scoring, not on downstream educational impact (e.g., student learning gains).
- Evaluation restricted to a single LLM (GPT-5.1) and domain (intermediate cybersecurity statistics), limiting generalizability across models or subjects.
- Dataset is proprietary and not publicly available, hindering external replication or benchmarking.
- No adversarial evaluation of evaluator robustness to intentionally misleading or low-quality questions is reported.
- Human rating sample size is moderate (60 per rubric) and may not cover full variability in question types and difficulty.
- Results do not compare extensively against alternative evaluation approaches beyond rubric-based LLM scoring modes.
Open questions / follow-ons
- How does improved human-machine alignment in rubric-based evaluation translate to actual improvements in learner outcomes when using AI-generated pretests?
- Can the rubric operationalization and rationale-first evaluation strategy generalize to other subject areas and educational levels?
- How do alternative evaluation frameworks, such as hybrid human-machine or contrastive AI models, compare in alignment and predictive validity?
- What are the effects of model updates (e.g., newer LLMs) or adversarial inputs on the reliability of automated pedagogical scoring?
Why it matters for bot defense
Bot defense practitioners developing CAPTCHA or evaluation systems that rely on automated quality assessments can benefit from the study’s insights on human-machine alignment for complex, multidimensional quality criteria. It emphasizes that ambiguities in rubric or decision criteria can produce systematic machine misjudgments, potentially allowing adversaries or low-quality content to slip through or be unfairly filtered. Integrating operationalized, explicit construct definitions with transparent machine reasoning (akin to rationale-first scoring) enhances alignment with human intentions. This suggests that deploying LLM-based evaluation modules for CAPTCHA quality control or bot detection should not rely solely on raw model outputs but incorporate detailed, domain-appropriate evaluation rubrics and intermediate rationale to improve interpretability and robustness.
Furthermore, the iterative generation–evaluation workflow balancing coverage across multiple perceptual dimensions offers a blueprint for developing diversified challenge sets. The findings caution against overreliance on generation capability alone without meticulous evaluation design, highlighting the importance of understanding systematic error patterns in machine judgment to harden systems against misclassification and improve defender confidence in automated filtering stages.
Cite
@article{arxiv2606_23629,
title={ Why Machines Misread Pedagogical Quality: Human-Machine Alignment in LLM-Based Pretest Question Evaluation },
author={ Pei-Yu Tseng and Mahir Akgun and Peng Liu },
journal={arXiv preprint arXiv:2606.23629},
year={ 2026 },
url={https://arxiv.org/abs/2606.23629}
}