Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach

Source: arXiv:2607.02432 · Published 2026-07-02 · By Manuel Alonso-Carracedo, Ruben Fernandez-Boullon, Pedro Celard, Francisco J. Rodriguez-Martinez, Lorena Otero-Cerdeira

TL;DR

This paper addresses the challenge of scalable and reliable grading of Linux/bash command-line examination responses, which are typically short text commands exhibiting multiple valid equivalences, partial correctness, and high syntactic sensitivity. The authors evaluate four state-of-the-art Large Language Models (LLMs)—GPT 5.2, Claude Opus 4.6, Gemini 3.0 Pro, and GLM 5—on 1200 real student exam responses labeled by three expert graders. They introduce a novel four-level cognitive taxonomy (CogTax) combining Bloom’s Revised Taxonomy cognitive complexity with an operational impact dimension tailored to Linux commands. The taxonomy allows fine-grained analysis of how grading agreement varies with question complexity. Using both minimal prompt and rubric-enhanced prompt variants, they find that rubric-guided prompts substantially improve human-AI grading agreement, with Gemini 3.0 Pro achieving the highest concordance (ICC(3,1) = 0.888, MAE = 0.10, Bland-Altman bias = −0.014). Agreement degrades consistently as question complexity rises from simple information queries (L1) to advanced system management tasks (L4). The study establishes the importance of rubric quality over model choice and proposes a principled framework to identify which commands are suitable for reliable AI-assisted grading versus requiring human review.

Key findings

• Gemini 3.0 Pro with rubric-enhanced prompts achieves highest agreement with human consensus: ICC(3,1) = 0.888, MAE = 0.10, Bland-Altman bias = −0.014.
• Agreement across all models declines as taxonomy level increases; largest discrepancies occur at Level 4 (advanced system management).
• Rubric-guided prompting substantially improves grading alignment compared to minimal prompting for all models.
• Rubric quality effects on agreement are larger than differences due to LLM provider/model choice.
• Three human graders show good to excellent inter-rater reliability with ICC(2,1) used as the reproducibility ceiling.
• The four-level cognitive taxonomy (CogTax) meaningfully stratifies grading difficulty for LLMs, linking grading errors to command complexity and operational impact.
• Bland-Altman plots show minimal systematic bias for best model (Gemini 3.0 Pro) and consistent limits of agreement with human grading.
• Using both domain-specific rubric and accepted correct solutions enables LLMs to better handle partial credit and syntactic variation.

Threat model

The adversary is modeled as the legitimate examiner and automated grading system assessing student command-line exam responses. The system assumes standard, non-adversarial exam conditions where student answers are truthful but varied in correctness and style. The adversary (the grader) knows the course content, the grading rubric, and model architectures but does not attempt to maliciously influence the evaluation. It cannot revise incorrect ground truth labels or access information beyond the exam data and rubric guidance.

Methodology — deep read

The study begins with a comprehensive threat model of grading Linux/bash command responses in educational exams, focusing on simulating expert human grading. The adversary model assumes standard exam conditions without cheating but considers the challenge of recognizing valid alternative solutions, partial correctness, and syntactic subtleties. Data comprises 1200 anonymized exam responses from second-year undergrad Computer Engineering students in a controlled OS course exam with 16 bash commands/exercises classified under a four-level CogTax taxonomy (L1: information query, L2: basic modifications, L3: structural understanding, L4: advanced system management). Three experienced instructors independently grade all responses using a detailed rubric specifying partial credit and common error deductions, enabling inter-rater reliability calculation (ICC(2,1)).

Four frontier LLMs (GPT 5.2, Claude Opus 4.6, Gemini 3.0 Pro, GLM 5) are evaluated independently on the same student responses. Each model is prompted twice: a minimal baseline prompt providing minimal context, and a rubric-enhanced prompt supplying the full grading rubric plus reference correct answers to guide scoring. Models output a numerical grade and justification per response.

Evaluation uses a multi-metric statistical battery capturing different agreement aspects: association metrics (Pearson and Spearman correlations), absolute agreement (ICC(2,1) for humans; ICC(3,1) for model-human concordance), error and bias (Mean Absolute Error, Bland-Altman analysis), and ordinal agreement (Weighted Kappa). These metrics are computed globally and stratified by the taxonomic level to analyze complexity effects.

A full human consensus score is computed as the average of the three human raters and used as ground truth for model comparisons. Agreement between each LLM and human consensus is assessed under both prompt variants, isolating the impact of rubric provision. Bland-Altman plots visualize systematic bias and variance across the score distribution.

No model fine-tuning or additional parameter adjustments are performed to avoid confounding; default model configurations ensure reproducibility. The experiment implements a blind grading interface for humans to avoid bias. The research platform managing data supports anonymization, question tagging, grader workflows, and detailed data export for analyses.

As a concrete example, a student response to a Level 3 question requiring a pipeline command is independently graded by three human instructors (scores averaged) and then submitted verbatim to Gemini 3.0 Pro with full rubric. The model produces a numerical score and explanation; agreement with human consensus is measured via ICC(3,1) and MAE. Across many such responses, overall grading quality is derived statistically.

Code and exact prompt templates are not explicitly stated to be released. The curriculum-specific taxonomy CogTax integrates Bloom’s cognitive levels with operational impact to tailor complexity measurement to Linux commands.

Technical innovations

• Integration of Bloom's Revised Taxonomy with an operational impact dimension to form a four-level, domain-specific cognitive taxonomy (CogTax) for Linux/bash command grading complexity.
• Systematic evaluation of four leading LLMs under controlled prompt ablation (minimal vs rubric-enhanced) on a large real-world dataset of 1200 student bash exam responses.
• Comprehensive multi-metric human-AI agreement framework combining ICC(2,1), ICC(3,1), Weighted Kappa, Bland-Altman analysis, and error metrics stratified by taxonomy level.
• Empirical demonstration that rubric prompt quality affects grading reliability more than choice of LLM provider or architecture, emphasizing structured prompt design.
• Proposal of a principled, taxonomy-based AI-assisted grading framework that identifies question domains where LLM grading is reliable versus requiring human intervention.

Datasets

• Linux/bash exam responses — 1200 student answers — collected from second-year Computer Engineering undergraduates in a controlled Operating Systems course exam

Baselines vs proposed

• Human inter-rater reliability (3 expert graders): ICC(2,1) = good to excellent reliability (exact values not specified).
• GPT 5.2 with minimal prompting: ICC(3,1) significantly lower than with rubric prompting (exact values not reported).
• Gemini 3.0 Pro with rubric-guided prompting: ICC(3,1) = 0.888, MAE = 0.10, Bland-Altman bias = −0.014; highest among evaluated models.
• Rubric quality impact on agreement > LLM provider/model impact on agreement (quantitative differences not explicitly tabulated but observed consistently).

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.02432.

Fig 1: Research methodology.

Fig 2: Used taxonomy to classify commands depending on its cognitive complexity and operational

Fig 3: Frequency distribution of normalized item-level scores assigned by human evaluators.

Fig 4: Distribution of assigned grade per taxonomy level for human evaluators.

Fig 5: Distribution of assigned grade per question for Variant 1 and Variant 2 across LLM models.

Fig 6: presents the performance scores obtained by GPT, Claude, Gemini, GLM eval-

Fig 7 (page 19).

Fig 8 (page 19).

Limitations

• The evaluation was limited to a single educational institution and course, potentially limiting generalizability to other curricula or domains.
• LLMs were used with default inference configurations without fine-tuning, which might not represent best achievable grading performance.
• No adversarial testing or robustness assessment against intentionally malformed or ambiguous responses was performed.
• The study focused on short-form Linux/bash commands; results may not transfer directly to longer coding assignments or non-shell domains.
• Exact prompt templates and evaluation platform are not publicly released, limiting reproducibility of the exact study pipeline.
• Taxonomy boundaries and rubric designs, while carefully constructed, are manually defined and may introduce subjective bias.

Open questions / follow-ons

• Can fine-tuning LLMs specifically on partial credit command-line grading data further improve reliability beyond rubric-prompting?
• How does the taxonomy-guided rubric framework generalize to other technical short-answer domains beyond Linux/bash?
• What is the impact of adversarial or ambiguous student inputs on LLM automated grading consistency?
• Can hybrid human-AI workflows dynamically route complex L3/L4 questions to human graders while automating simpler L1/L2 reliably?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this work highlights the value of a structured, taxonomy-based complexity framework to predict where AI evaluation is reliable or prone to error, analogous to assessing when automated systems can distinguish valid human input from bots in variable command contexts. The paper demonstrates that prompt engineering—providing explicit rubrics or structured guidance—significantly improves AI judgment accuracy, a lesson transferable to designing challenge-response systems where interpretability and exactness of AI decisions matter. Furthermore, stratifying tasks by cognitive and operational complexity can inform adaptive defenses that escalate suspicious or complex interactions for human review. However, limitations on AI error at higher complexity levels caution practitioners against fully autonomous AI grading or assessment without fallback human oversight.

Cite

@article{arxiv2607_02432,
  title={ Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach },
  author={ Manuel Alonso-Carracedo and Ruben Fernandez-Boullon and Pedro Celard and Francisco J. Rodriguez-Martinez and Lorena Otero-Cerdeira },
  journal={arXiv preprint arXiv:2607.02432},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.02432}
}

Read the full paper

• arXiv:2607.02432 (abstract)
• arXiv:2607.02432 (PDF)