Human-in-the-Loop Benchmarking of Heterogeneous LLMs for Automated Competency Assessment in Secondary Level Mathematics

2026-04-29 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

A "Human-in-the-Loop" benchmarking framework was developed to assess the effectiveness of multiple Large Language Models (LLMs) in automating secondary-level mathematics assessment, specifically for Grade 10 Optional Mathematics in Nepal. The framework utilized a multi-dimensional rubric covering four topics and four competencies: Comprehension, Knowledge, Operational Fluency, and Behavior and Correlation. An ensemble of open-weight models (Eagle Llama 3.1-8B, Orion Llama 3.3-70B) and proprietary frontier models (Nova Gemini 2.5 Flash, Lyra Gemini 3 Pro) was benchmarked against a ground truth established by two senior mathematics faculty members (kappa_w = 0.8652). Findings revealed an "Architecture-compatibility gap," where Gemini-based Mixture-of-Experts (Sparse MoE) models achieved "Fair Agreement" (kappa_w ~ 0.38), but the larger Orion (70B) model showed "No Agreement" (kappa_w = -0.0261). This suggests that architectural compliance with instruction constraints is more critical than raw parameter scale for rubric-constrained tasks.

Key takeaway

For educators considering LLMs for competency-based assessment, prioritize models with architectures known for instruction compliance, such as Sparse MoE, over simply larger models. While LLMs are not yet ready for autonomous certification, integrating them into a "Human-in-the-Loop" framework can provide valuable assistive support for preliminary evidence extraction, significantly reducing manual effort in qualitative competency mapping.

Key insights

LLM architecture compatibility with instruction constraints is more critical than model scale for rubric-constrained assessment tasks.

Principles

• Architecture compatibility outweighs parameter scale.
• Human-in-the-loop improves LLM assessment reliability.

Method

A "Human-in-the-Loop" benchmarking framework assesses LLMs using a multi-dimensional rubric and compares against human-defined ground truth to evaluate agreement.

In practice

• Use Sparse MoE models for rubric-based tasks.
• Integrate human oversight for LLM-based assessments.

Topics

• Competency-Based Education
• LLM Benchmarking
• Secondary Mathematics Assessment
• Human-in-the-Loop
• LLM Architectures

Best for: AI Scientist, Research Scientist, AI Student

Related on AIssential

• LLM-as-Judge in Education: A Curriculum-Grounded Marking Pipeline — LLM assessment grounded in official curriculum artifacts improves consistency, transparency, and traceability for high-stakes education.
• Small, Private Language Models as Teammates for Educational Assessment Design — SLMs offer competitive, privacy-sensitive assessment design, but require human oversight due to model evaluation biases.
• Engineering a human-aligned LLM evaluation workflow with Prodigy and DSPy — Human-aligned LLM evaluation requires iterative workflows that integrate granular human feedback to engineer context-specific metrics.
• The Architecture Behind Open-Source LLMs — MoE architectures and diverse training strategies drive rapid progress in open-weight LLMs, optimizing cost and performance.
• How Uncertain Is the Grade? A Benchmark of Uncertainty Metrics for LLM-Based Automatic Assessment — Categorical uncertainty metrics are generally more effective for LLM-based automatic assessment, while relation-based methods offer superior stability.
• GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theory — LLMs struggle with mathematical reasoning, particularly graduate-level proofs, with GPT-5 significantly outperforming other models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.