Caraman at SemEval-2026 Task 8: Three-Stage Multi-Turn Retrieval with Query Rewriting, Hybrid Search, and Cross-Encoder Reranking

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

Caraman and Silaghi's system for SemEval-2026 Task 8 (MTRAGEval), specifically Task A (Retrieval) across four English-language domains, employs a three-stage pipeline. The first stage involves query rewriting using a LoRA-finetuned Qwen 2.5 7B model to transform context-dependent follow-up questions into standalone queries. This is followed by a hybrid search stage combining BM25 and dense retrieval, integrated through Reciprocal Rank Fusion. The final stage utilizes BGE-reranker-v2-m3 for cross-encoder reranking. On the official test set, the system achieved an nDCG@5 of 0.531, securing 8th place among 38 participating systems and outperforming the organizer baseline by 10.7%. Development comparisons also revealed that domain-specific temperature tuning for query generation, favoring deterministic decoding for technical domains and controlled randomness for general domains, consistently improved performance, while more complex strategies like domain-aware prompting degraded it.

Key takeaway

For Machine Learning Engineers building multi-turn retrieval systems, adopting a structured three-stage pipeline can significantly enhance performance. You should consider implementing query rewriting with a LoRA-finetuned Qwen 2.5 7B, integrating hybrid BM25 and dense retrieval via Reciprocal Rank Fusion, and applying BGE-reranker-v2-m3 for reranking. Crucially, experiment with domain-specific temperature tuning for query generation, as this simple optimization offers consistent gains over more complex approaches.

Key insights

A three-stage pipeline combining query rewriting, hybrid search, and cross-encoder reranking significantly improves multi-turn retrieval performance.

Principles

• Domain-specific temperature tuning optimizes query generation.
• Simpler, targeted strategies can outperform complex ones.
• Reciprocal Rank Fusion effectively merges hybrid search results.

Method

The method involves LoRA-finetuned Qwen 2.5 7B for query rewriting, followed by BM25 and dense retrieval combined with Reciprocal Rank Fusion, and finally BGE-reranker-v2-m3 for cross-encoder reranking.

In practice

• Apply LoRA-finetuning to Qwen 2.5 7B for query rewriting.
• Integrate BM25 and dense retrieval using Reciprocal Rank Fusion.
• Utilize BGE-reranker-v2-m3 for reranking search results.

Topics

• Multi-Turn Retrieval
• Query Rewriting
• Hybrid Search
• Cross-Encoder Reranking
• Qwen 2.5 7B
• Reciprocal Rank Fusion

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Advanced Retrieval Pipeline for RAG (HyDE, Hybrid Search, Reranking) | Build 100% Local Retrieval — Combining HyDE, hybrid search, and reranking significantly enhances RAG retrieval accuracy and relevance.
• Optimizing Efficiency in Multi-Stage Semantic Re-ranking Architectures — Cascade re-ranking significantly reduces latency in legal IR with minimal precision loss.
• What is reciprocal rank fusion in hybrid search? — Reciprocal Rank Fusion effectively combines diverse ranker outputs into a robust global ranking without training.
• Not your usual search system — feat. BM25, FAISS and XGBoost — Production-grade search systems combine lexical and semantic retrieval with learning-to-rank for optimal performance.
• 3x Faster Search: Parallel Test-Time Scaling with Instructed-Retriever-1 — Parallel test-time scaling with a specialized model significantly boosts retrieval speed and quality by executing search stages concurrently.
• KaLM-Reranker-V1: Fast but Not Late Interaction for Compressed Document Reranking — KaLM-Reranker-V1 decouples query and passage encoding for efficiency while using cross-attention for expressive relevance modeling.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.