IITKanBDone at SemEval-2026 Task 8: MTRAGEval - Evaluating Multi-Turn RAG Conversations

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

IITKanBDone's system for SemEval-2026 Task 8, named MTRAGEval, addresses multi-turn conversational question answering using retrieval-augmented generation (RAG). The team participated in three sub-tasks: Task A (retrieval), Task B (generation with reference passages), and Task C (end-to-end RAG). For Task A, ELSER (Elastic Learned Sparse EncodeR) achieved the best retrieval performance with an nDCG@5 of 0.4018, ranking 24/38 among evaluated methods like BM25 and BGE. Task B utilized the Mistral-7B-Instruct-v0.2 model via HuggingFace for response generation, securing a harmonic mean score of 0.6976 (Rank 13/26). The end-to-end Task C combined ELSER retrieval with Mistral-7B generation, using top-5 retrieved passages, resulting in a score of 0.4289 (Rank 23/29). This system highlights the efficacy of learned sparse retrieval and instruction-tuned models in multi-turn RAG scenarios.

Key takeaway

For NLP Engineers developing multi-turn conversational AI systems, this work suggests prioritizing learned sparse retrieval methods like ELSER and instruction-tuned models such as Mistral-7B-Instruct-v0.2. You should benchmark these components individually and in an end-to-end RAG pipeline, focusing on metrics like nDCG@5 for retrieval and harmonic mean for generation, to optimize performance in complex conversational QA scenarios.

Key insights

Learned sparse retrieval and instruction-tuned models effectively enhance multi-turn RAG conversational question answering.

Principles

• ELSER excels in sparse retrieval for multi-turn RAG.
• Instruction-tuned models like Mistral-7B perform well in RAG generation.

Method

Combine ELSER for top-5 passage retrieval with Mistral-7B-Instruct-v0.2 for response generation in multi-turn RAG scenarios.

In practice

• Evaluate ELSER for retrieval in conversational RAG systems.
• Consider Mistral-7B-Instruct-v0.2 for RAG response generation.

Topics

• Multi-Turn RAG
• Conversational AI
• Retrieval-Augmented Generation
• SemEval
• ELSER
• Mistral-7B

Best for: AI Engineer, Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• Your RAG Pipeline Is Probably Useless. Here’s a Better Alternative — RAG pipelines fail predictably; alternatives like long-context, summarization, structured, or graph-based retrieval offer better solutions.
• Generative AI-Based Virtual Assistant using Retrieval-Augmented Generation: An evaluation study for bachelor projects — RAG-based virtual assistants with self-reflection can effectively address LLM limitations in specialized academic contexts.
• A Comparative Study of Language Models for Khmer Retrieval-Augmented Question Answering — RAG efficacy for low-resource languages like Khmer is bottlenecked by retriever performance, with generator strengths varying across RAGAS metrics.
• Retrieval is Cheap, Show Me the Code: Executable Multi-Hop Reasoning for Retrieval-Augmented Generation — Reformulating multi-hop RAG as program synthesis and execution enhances verifiability and performance.
• Building a PDF Question-Answering Chatbot with Spring AI: From PDF Upload to RAG-Powered Answers — RAG systems enhance LLM accuracy by grounding answers in retrieved, relevant private data, outperforming larger models without context.
• Why Retrieval-Augmented Generation Fails: A Graph Perspective — RAG failures stem from shallow, fragmented internal information flow, not just poor retrieval.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.