Toward Cross-Domain Automated Feedback: A Comparative Evaluation of Open-Source Models across Diverse Student Assessment Types

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Education & Learning — Educational Technology (EdTech), Artificial Intelligence & Machine Learning · Depth: Advanced, medium

Summary

A study evaluated the potential of two open-source large language models, DeepSeek R1 and Qwen 3.5, for automated feedback generation across diverse student assessment types. Researchers compared their performance on programming assignments, essays, and mathematics problems, utilizing both unified and multi-agent prompting strategies. The analysis, assessed by human judgment criteria, aimed to identify scalable alternatives to proprietary LLM-based feedback systems for educational settings. Results indicate that a single open-source model can provide useful feedback across domains, though with varying effectiveness. DeepSeek R1 showed stronger performance on reasoning-intensive tasks like mathematics, while Qwen 3.5 excelled in holistic tasks such as writing. Both models struggled with programming. The study concluded that model architecture is more influential than prompting strategy, and a multi-agent approach does not consistently improve results over a unified LLM.

Key takeaway

For educational technologists or AI scientists developing automated feedback systems, consider open-source LLMs like DeepSeek R1 or Qwen 3.5 as cost-effective alternatives to proprietary APIs. You should align model choice with task type: DeepSeek R1 for structured, reasoning-intensive subjects like mathematics, and Qwen 3.5 for holistic tasks such as writing. Be aware that both models currently struggle with programming assignments, and a multi-agent prompting approach may not consistently yield better results than a simpler unified strategy.

Key insights

Open-source LLMs can provide cross-domain automated feedback, with model architecture influencing task suitability.

Principles

• Model architecture outweighs prompting strategy for feedback quality.
• Reasoning-optimized LLMs suit structured tasks.
• General-purpose LLMs perform better on holistic tasks.

Method

The study evaluated DeepSeek R1 and Qwen 3.5 on programming, essays, and math using unified and multi-agent prompting, assessed by human judgment.

In practice

• Consider DeepSeek R1 for math-heavy feedback.
• Use Qwen 3.5 for essay or writing feedback.
• Expect challenges with programming assignment feedback.

Topics

• Automated Feedback
• Open-Source LLMs
• DeepSeek R1
• Qwen 3.5
• Educational Technology
• Prompting Strategies
• Student Assessment

Best for: AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• The Architecture Behind Open-Source LLMs — MoE architectures and diverse training strategies drive rapid progress in open-weight LLMs, optimizing cost and performance.
• Qwen 3.5 The GREATEST Opensource AI Model That Beats Opus 4.5 and Gemini 3? (Fully Tested) — Qwen 3.5 is a powerful open-source, multimodal AI model excelling in speed and diverse applications.
• Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most — LLM tutoring agents excel at optimal step identification but struggle with nuanced diagnostic feedback and instructional effectiveness.
• Paper: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek R1 enhances LLM reasoning via pure reinforcement learning and rule-based rewards, fostering autonomous problem-solving without supervised data.
• LLM-based Models for Detecting Emerging Topics in Service Feedback — Integrating fine-tuned, quantized LLMs with human expertise reliably detects emerging service quality topics and potential inequities in feedback.
• GLM 5.1 Agentic Coding Test with OpenCode | New Best Open Coding LLM? | Live Test — GM 5.1 excels in agentic coding and project understanding, outperforming some established models in specific benchmarks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.