CAFES: A Collaborative Multi-Agent Framework for Multi-Granular Multimodal Essay Scoring

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing & Understanding · Depth: Expert, medium

Summary

CAFES, a collaborative multi-agent framework, addresses limitations in Automated Essay Scoring (AES), particularly for multimodal assessments. Traditional AES methods lack generalizability and multimodal perception, while even recent Multimodal Large Language Model (MLLM)-based approaches often generate hallucinated justifications and scores misaligned with human judgment. Developed by Jiamin Su et al. for the 4th Workshop on Advances in Language and Vision Research (ALVR) in July 2026, CAFES orchestrates three specialized agents: an Initial Scorer for rapid, trait-specific evaluations; a Feedback Pool Manager to aggregate detailed, evidence-grounded feedback; and a Reflective Scorer that iteratively refines scores to enhance human alignment. Experiments using widely adopted MLLMs demonstrated an average relative improvement of 21% in Quadratic Weighted Kappa (QWK) against ground truth, with significant gains in grammatical and lexical diversity. The code and dataset are publicly available.

Key takeaway

For AI Scientists or Machine Learning Engineers developing Automated Essay Scoring (AES) systems for multimodal content, you should consider adopting a collaborative multi-agent framework. This approach, exemplified by CAFES, significantly boosts evaluation generalizability and human alignment, achieving a 21% relative improvement in Quadratic Weighted Kappa. Integrating specialized agents and iterative feedback loops can effectively mitigate hallucination issues and score misalignment often seen in direct Multimodal Large Language Model applications, leading to more robust and trustworthy assessment tools.

Key insights

CAFES, a multi-agent framework, significantly improves multimodal essay scoring accuracy and human alignment by orchestrating specialized agents.

Principles

• Collaborative multi-agent design enhances complex task performance.
• Iterative refinement improves alignment with human judgment.
• Specialized agents handle distinct evaluation aspects.

Method

CAFES orchestrates an Initial Scorer for rapid evaluation, a Feedback Pool Manager for evidence-grounded feedback, and a Reflective Scorer that iteratively refines scores based on this feedback to enhance human alignment.

In practice

• Apply multi-agent systems for complex evaluations.
• Integrate iterative feedback loops for score refinement.
• Utilize MLLMs within structured agent frameworks.

Topics

• Automated Essay Scoring
• Multimodal AI
• Multi-agent Systems
• Multimodal Large Language Models
• Natural Language Processing

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• CAF-Gen: A Multi-Agent System for Enriching Argumentation Structures — CAF-Gen's multi-agent Creator-Reviewer system enriches shallow argument structures into robust, CAF-compliant models, overcoming single-pass generation instability.
• AMA: Adaptive Memory via Multi-Agent Collaboration — AMA uses multi-agent collaboration and hierarchical memory to provide adaptive, consistent long-term memory for LLM agents.
• MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights — Multilingual LLM evaluation requires fine-grained, agentic diagnosis to transform scores into actionable insights, addressing cultural and deployment realities.
• Improving Collaborative Storytelling with a Multi-Agent Framework Based on Large Language Models — An iterative LLM Writer-Editor framework significantly improves collaborative storytelling quality, even with few refinement steps.
• Leveraging Multimodal Self-Consistency Reasoning in Coding Motivational Interviewing for Alcohol Use Reduction — Multimodal self-consistency reasoning significantly enhances automated Motivational Interviewing (MI) coding by integrating diverse audio-language model predictions.
• MADRAG: Multi-Agent Debate with Retrieval-Augmented Generation for Training-Free Analytic Essay Scoring — MADRAG employs multi-agent debate and retrieval-augmented generation for training-free, reliable analytic essay scoring, surpassing prompt-based LLM baselines.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.