Evaluating LLM-Generated Formative Feedback for Undergraduate Mathematics Through the Lens of Feedback Theory

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, AI in Education · Depth: Advanced, quick

Summary

A study evaluated large language model-generated feedback on 65 undergraduate proof-writing exercises, comparing GPT-4.1 and GPT-5 under two workflow configurations. Researchers used Hattie and Timperley's feedback framework and a grade agreement metric, with two independent LLM evaluators. A mark-scheme-augmented workflow significantly improved grade correlation with human experts for both models, also enabling instructors to audit the system pre-deployment. GPT-5 consistently produced higher-quality feedback across all evaluated dimensions. The collected metrics suggest high feedback quality in this specific setting, a finding supported by several experimental sanity checks. However, further work is needed to provide meaningful self-regulation support and conduct controlled tests with students.

Key takeaway

For research scientists developing AI-driven educational feedback systems, you should prioritize integrating mark-scheme-augmented workflows to enhance feedback quality and enable pre-deployment auditing. Your evaluation should leverage established feedback frameworks like Hattie and Timperley's, as this study demonstrates their utility. Consider advanced models like GPT-5, which showed superior performance, to maximize feedback effectiveness for complex subjects like undergraduate mathematics proof-writing.

Key insights

LLM-generated feedback for math proofs can be high-quality, especially with structured workflows and advanced models like GPT-5.

Principles

• Feedback theory offers a useful evaluation lens.
• Workflow design impacts LLM feedback quality.
• Auditing LLM systems pre-deployment is crucial.

Method

Evaluated LLM feedback on 65 undergraduate proof-writing exercises using Hattie and Timperley's framework and grade agreement, comparing GPT-4.1 and GPT-5 with two workflows.

In practice

• Augment LLM workflows with mark schemes.
• Prioritize GPT-5 for feedback generation.
• Use feedback theory for evaluation.

Topics

• LLM Feedback
• Formative Assessment
• Undergraduate Mathematics
• Proof Writing
• GPT-5
• Feedback Theory

Best for: AI Scientist, Research Scientist

Related on AIssential

• Evaluation of LLMs for Mathematical Formalization in Lean — Iterative refinement significantly boosts LLM performance in formal mathematical proof generation.
• An Explainable AI Assistant for Introductory Programming Education: Improving Feedback Reliability with Instructor-AI Collaboration — Instructor-AI collaboration with explainable models and constrained LLMs provides scalable, reliable, and pedagogically sound programming feedback.
• LLM-as-Judge in Education: A Curriculum-Grounded Marking Pipeline — LLM assessment grounded in official curriculum artifacts improves consistency, transparency, and traceability for high-stakes education.
• GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theory — LLMs struggle with mathematical reasoning, particularly graduate-level proofs, with GPT-5 significantly outperforming other models.
• More than a Judge: An Empirical Study of Agent-Human Interaction in Crowdsourced Testing Assessment — Agent-generated feedback, derived from LLM-based assessment, significantly improves crowdsourced test report quality and fosters skill transfer.
• Measuring Whether LLM Tutors Teach or Solve: A Diagnostic for Educational Impact — LLM tutor effectiveness requires evaluating pedagogical support beyond mere task-solving ability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.