RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generation

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, medium

Summary

RaguTeam secured first place in Task B of the SemEval-2026 Task 8 MTRAGEval shared task, focusing on multi-turn retrieval-augmented generation. Their submission utilized a heterogeneous ensemble of seven Large Language Models, organized into two groups with distinct prompting strategies. A GPT-4o-mini judge then selected the optimal response from candidates. The system achieved a conditioned harmonic mean score of 0.78, significantly surpassing the strongest organizer baseline of 0.64, and ranked first among 26 competing teams. Ablation studies confirmed that diversity across model families, scales, and prompting approaches is crucial for ensemble performance. The team also integrated Meno-Lite-0.1, a 7B domain-adapted model, noted for its favorable cost-performance trade-off, and provided an analysis of MTRAGEval's annotation limitations.

Key takeaway

For AI scientists and ML engineers building multi-turn RAG systems, consider implementing a judge-orchestrated LLM ensemble to enhance response faithfulness and accuracy. Your system's performance can significantly improve by leveraging diverse LLM families and prompting strategies, as demonstrated by RaguTeam's first-place SemEval-2026 result. Additionally, explore integrating smaller, domain-adapted models like Meno-Lite-0.1 to optimize your cost-performance balance without sacrificing quality.

Key insights

A judge-orchestrated, diverse LLM ensemble significantly improves multi-turn RAG performance over individual models.

Principles

• Diversity across LLM families, scales, and prompts is critical.
• Ensembles consistently outperform individual models in RAG.

Method

A heterogeneous ensemble of seven LLMs, split into two groups with distinct prompting strategies, generates candidate responses, with a GPT-4o-mini judge selecting the best output.

In practice

• Employ GPT-4o-mini as an LLM response judge.
• Integrate 7B domain-adapted models for cost efficiency.
• Vary LLM families and prompting for ensemble diversity.

Topics

• LLM Ensembles
• Retrieval-Augmented Generation
• SemEval-2026 Task 8
• GPT-4o-mini
• Multi-Turn Response Generation
• Meno-Lite-0.1

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Büyük Dil Modellerinin Hafızasını Güçlendiren Yaklaşım:Retrieval-Augmented Generation (RAG) — RAG enhances LLMs by providing external, real-time context, reducing hallucinations and improving factual accuracy.
• Consistent and Distinctive: LLM Benchmark Efficiency via Maximum Independent Set Prompt Selection on Similarity Graphs — A graph-based Maximum Independent Set (MIS) approach efficiently selects diverse LLM benchmark prompts, maintaining ranking consistency.
• Nous: An Attempt to Extract and Inject the Cognition Behind Prediction-Market Behavior — LLM agent ensembles face cognitive monoculture; prompt-level injection of human behavioral profiles fails to induce diversity.
• From LLMs to Products: Alignment & Production — Transforming base LLMs into reliable, cost-effective production systems requires both alignment and deployment optimization.
• True Zero-shot MT — LLM evaluation faces a benchmark crisis, while specific models like Command R+ excel in RAG and multilingual tasks.
• [NOTICE] This feed is no longer maintained — The Mastering LLMs conference provides practical, unbiased, and diverse expert insights into LLM deployment and fine-tuning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.