uircis at SemEval-2026 Task 8: A Unified Lightweight Pipeline for Multi-Turn RAG Evaluation

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, NLP & Computational Linguistics · Depth: Expert, short

Summary

The "uircis" system, submitted for SemEval-2026 Task 8 (MTRAGEval), presents a unified lightweight pipeline designed for multi-turn RAG evaluation, addressing both retrieval (Subtask A) and generation (Subtask B). This fully reproducible approach utilizes open-weight models, specifically Qwen2.5-7B-Instruct for query rewriting and grounded answer generation, BGE-M3 for dense retrieval, and BGE-Reranker-v2-M3 for cross-encoder reranking. The authors report official test performance and detail ablation experiments that quantify the impact of both rewriting and reranking across various domains. Furthermore, the paper includes an error analysis, leveraging the organizers' analytics and answerability classes, to pinpoint key failure modes related to multi-turn retrieval specificity and grounded generation.

Key takeaway

For Machine Learning Engineers building multi-turn RAG systems, this work provides a validated open-weight pipeline architecture. You should consider integrating Qwen2.5-7B-Instruct for query rewriting and generation, paired with BGE-M3 for retrieval and BGE-Reranker-v2-M3 for reranking, to establish a reproducible baseline. Analyze your system's error modes, particularly multi-turn retrieval specificity and grounded generation, to guide targeted improvements.

Key insights

The paper details a lightweight, reproducible multi-turn RAG pipeline using specific open-weight models for evaluation.

Principles

• Open-weight models can form robust RAG pipelines.
• Ablation studies quantify component impact.
• Error analysis reveals specific RAG failure modes.

Method

The pipeline uses Qwen2.5-7B-Instruct for query rewriting and generation, BGE-M3 for dense retrieval, and BGE-Reranker-v2-M3 for reranking in a multi-turn RAG setup.

In practice

• Implement Qwen2.5-7B-Instruct for RAG.
• Use BGE-M3 for dense retrieval.
• Integrate BGE-Reranker-v2-M3 for reranking.

Topics

• Multi-turn RAG
• RAG Evaluation
• Open-weight Models
• Qwen2.5-7B-Instruct
• BGE-M3
• Cross-encoder Reranking

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Issue #115 - Reranking in your RAG pipeline — Reranking enhances RAG pipeline precision by reordering initially retrieved documents using a Cross-Encoder.
• Top 5 Embedding Models for Your RAG Pipeline — Effective RAG pipelines rely on embedding models that balance retrieval performance, real-world adoption, and deployment practicality.
• Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models — RAG-based vulnerability detection with open-weight LLMs faces a performance plateau around 0.30 pairwise accuracy.
• Building a Production-Ready RAG Pipeline — Only 5 Seats Left — The workshop focuses on advanced techniques for building production-ready Retrieval Augmented Generation (RAG) pipelines.
• OpenRAG: An open-source stack for RAG — Phil Nash — OpenRAG is an open-source, agentic RAG stack combining Docling, OpenSearch, and LangFlow for flexible, customizable retrieval-augmented generation.
• Vector Search Using Ollama for Retrieval-Augmented Generation (RAG) — RAG combines vector search with LLMs to provide context-aware, fact-grounded responses from external data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.