RAGonauts at SemEval-2026 Task 8: BM25 Retrieval with Last-Turn Query Formulation for Multi-Turn RAG Conversations

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

The RAGonauts team's submission to SemEval-2026 Task 8: MTRAGEval presents a lightweight, training-free retrieval framework for multi-turn Retrieval-Augmented Generation (RAG) conversations. This system, designed for Task A, addresses the challenge of retrieving relevant passages from conversational history where user queries often contain implicit references. It employs BM25 ranking combined with conversational query formulation, deriving queries from dialogue turns and utilizing domain-specific indices to maintain corpus relevance. The RAGonauts system achieved an nDCG@5 score of 0.2836 on the official evaluation set, securing the 33\textsuperscript{rd} rank. This performance, achieved without neural retrievers or fine-tuning, highlights that sparse lexical retrieval remains an efficient and reproducible baseline for developing conversational RAG systems.

Key takeaway

For NLP Engineers building multi-turn RAG systems, consider BM25 retrieval with last-turn query formulation as a robust, training-free baseline. This approach, demonstrated by a 0.2836 nDCG@5 score at SemEval-2026 Task 8, offers an efficient and reproducible starting point. You can achieve competitive performance without complex neural models or extensive fine-tuning, streamlining development and deployment.

Key insights

BM25 with last-turn query formulation offers an efficient, training-free baseline for multi-turn RAG retrieval.

Principles

• Sparse lexical retrieval is an efficient baseline.
• Conversational history informs query formulation.
• Domain-specific indices preserve corpus relevance.

Method

The system uses BM25 ranking with queries derived from dialogue turns, performing retrieval via domain-specific indices without neural retrievers or fine-tuning.

In practice

• Implement BM25 for initial RAG retrieval.
• Formulate queries from last-turn dialogue.
• Utilize domain-specific knowledge bases.

Topics

• Multi-turn RAG
• BM25 Retrieval
• Conversational AI
• Query Formulation
• Semantic Evaluation
• Information Retrieval

Best for: AI Engineer, Machine Learning Engineer, Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Your RAG Pipeline Is Probably Useless. Here’s a Better Alternative — RAG pipelines fail predictably; alternatives like long-context, summarization, structured, or graph-based retrieval offer better solutions.
• META’s New SIRA: Superintelligence RAG — SIRA uses an LLM's parametric memory and BM25 for deterministic, single-shot information retrieval.
• Hybrid Search for RAG: BM25 + Vectors (When Each Wins) — Effective RAG systems require hybrid search, combining lexical and semantic retrieval, to prevent retrieval failures.
• Stop Saying RAG Is Dead — This article argues that "RAG is not dead," but rather the oversimplified 2023 approach of stuffing documents into a vector database and using cosine similarity is flawed due to critical information loss.
• Understanding the Behaviors of Environment-aware Information Retrieval — LLMs can learn to adapt query strategies for diverse retrievers using RL, revealing distinct optimal query styles.
• Beyond Semantic Search: The Complete Guide to Advanced RAG with Milvus | Sagar Patil — Advanced RAG requires multi-stage retrieval, combining hybrid search with techniques like query rewriting and reranking for production reliability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.