AILS-NTUA at SemEval-2026 Task 8: Evaluating Multi-Turn RAG Conversations

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, medium

Summary

The AILS-NTUA system, presented at SemEval-2026 Task 8 (MTRAGEval), addresses multi-turn retrieval-augmented generation across passage retrieval (A), reference-grounded response generation (B), and end-to-end RAG (C). Its design employs a query-diversity-over-retriever-diversity strategy, utilizing multiple LLM-based query reformulations with a single sparse retriever and a variance-aware nested Reciprocal Rank Fusion. Additionally, an agentic generation pipeline handles grounded response generation through evidence span extraction, dual-candidate drafting, and calibrated multi-judge selection. The system achieved strong results, ranking first in Task A and second in Task B. Empirical findings suggest that query diversity with a well-aligned retriever outperforms heterogeneous retriever ensembling, and answerability calibration is the primary bottleneck for end-to-end performance, not retrieval coverage.

Key takeaway

For Machine Learning Engineers developing multi-turn RAG systems, prioritize enhancing query diversity through LLM-based reformulations rather than complex retriever ensembles. Focus development efforts on robust answerability calibration within your generation pipeline, as this is a critical bottleneck for end-to-end performance. Implementing an agentic generation approach with calibrated multi-judge selection can significantly improve response quality and system reliability.

Key insights

Query diversity and answerability calibration are critical for effective multi-turn RAG performance.

Principles

• Prioritize query diversity over retriever diversity.
• Decompose generation into agentic sub-tasks.
• Calibrate answerability for end-to-end RAG.

Method

The system uses LLM-based query reformulations with a sparse retriever and Reciprocal Rank Fusion. Grounded response generation involves evidence span extraction, dual-candidate drafting, and multi-judge selection.

In practice

• Implement LLM-based query reformulation.
• Use Reciprocal Rank Fusion for retrieval.
• Integrate multi-judge selection for responses.

Topics

• Retrieval-Augmented Generation
• Multi-Turn Conversations
• Query Reformulation
• Semantic Evaluation
• Answerability Calibration
• Agentic Generation

Best for: Research Scientist, AI Architect, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• An Efficient Approach for Answering Not Readily Attainable Questions for RAG-based Applications — This paper proposes an efficient approach for RAG systems to answer questions not readily found in retrieved data.
• MADRAG: Multi-Agent Debate with Retrieval-Augmented Generation for Training-Free Analytic Essay Scoring — MADRAG employs multi-agent debate and retrieval-augmented generation for training-free, reliable analytic essay scoring, surpassing prompt-based LLM baselines.
• Building a Production-Ready Agentic Chatbot Course [3/30]: Designing Production-Ready RAG Pipeline — A production RAG pipeline requires a multi-stage approach beyond simple vector search to ensure reliable information retrieval.
• How to Teach the LLM to Think With Your Data — Feeding RAG-retrieved context into an LLM enables reasoning, reducing hallucinations and producing more natural, coherent answers.
• Agent-Orchestrated Adaptive RAG: A Comparative Study on Structured and Multi-Hop Retrieval — Agentic RAG enhancements must be selectively applied based on query and domain characteristics for optimal performance.
• RCEM: Robust Conversational Search EMbedder in Distributional Shift — RCEM distills LLM query rewriting into an embedder for robust, low-latency conversational search without explicit rewriting.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.