Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach

2026-07-02 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

A study evaluated four frontier Large Language Models (GPT, Claude Opus, Gemini, and GLM) for automated grading of short Linux/bash command responses, addressing challenges in computing education due to rising enrolments. The research employed a four-level cognitive taxonomy, combining cognitive complexity and operational impact, from information retrieval (L1) to advanced system management (L4). Models were tested with minimal and rubric-enhanced prompts on 1200 real responses from second-year Computer Engineering students, previously graded by three expert instructors. Gemini 3.0 Pro with rubric-guided prompting achieved the highest human-AI agreement (ICC(3,1) = 0.888, MAE = 0.10, Bland-Altman bias = -0.014). Agreement consistently declined as the taxonomy level increased, with larger discrepancies at higher complexity. Rubric quality had a greater impact than provider choice, consistently improving agreement across all models. This establishes a taxonomy-based framework for AI-assisted grading suitability.

Key takeaway

For AI Engineers developing automated grading systems in computing education, you should prioritize robust rubric design over specific LLM selection. Implement a cognitive taxonomy to identify questions suitable for AI-assisted grading, reserving complex L3 and L4 structural or system management tasks for human review. This approach ensures reliable assessment while leveraging LLMs for efficiency in lower-complexity tasks.

Key insights

LLMs can reliably grade Linux/bash exams, especially with rubrics, but struggle with higher cognitive complexity questions.

Principles

• Rubric quality outweighs LLM choice for grading accuracy.
• Question complexity predicts LLM grading difficulty.
• Taxonomy-based frameworks guide AI-assisted grading.

Method

Evaluate LLMs for grading using a four-level cognitive taxonomy, comparing minimal vs. rubric-enhanced prompts against expert human scores on real student responses.

In practice

• Use rubric-guided prompts for LLM-based grading.
• Prioritize human review for L3/L4 questions.
• Apply cognitive taxonomy to assess LLM grading suitability.

Topics

• Large Language Models
• Automated Grading
• Computing Education
• Linux/bash
• Cognitive Taxonomy
• LLM Evaluation

Best for: AI Scientist, Research Scientist, AI Engineer

Related on AIssential

• Grading the Grader: Lessons from Evaluating an Agentic Data Analysis System — Evaluating agentic data analysis systems requires robust, multi-layered grading strategies to accurately distinguish genuine disagreements from grading artifacts.
• LLM-as-Judge in Education: A Curriculum-Grounded Marking Pipeline — LLM assessment grounded in official curriculum artifacts improves consistency, transparency, and traceability for high-stakes education.
• CogRAG+: Cognitive-Level Guided Diagnosis and Remediation of Memory and Reasoning Deficiencies in Professional Exam QA — Aligning RAG with human cognitive hierarchies significantly boosts LLM performance on professional exams without fine-tuning.
• Beyond Grading Accuracy: Exploring Alignment of TAs and LLMs — Open-source LLMs can effectively grade UML class diagrams with high per-criterion accuracy, aiding TAs.
• Creating and Evaluating K-12 GenAI Assessment Graders Through Context Engineering — Context engineering enables LLMs to reliably grade K-12 assessments, particularly for objective subjects, but human oversight is vital.
• Characterising LLM-Generated Competency Questions: a Cross-Domain Empirical Study using Open and Closed Models — LLMs can automate Competency Question generation, but their output quality varies significantly by model and use case.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.