NLP-CEIA-UFG at SemEval-2026 Task 8: Iterative Retrieval with Notes-Guided Query Refinement for Multi-Turn RAG

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, medium

Summary

The NLP-CEIA-UFG system, developed for SemEval-2026 Task 8, addresses multi-turn retrieval-augmented generation (RAG) across diverse document corpora. Its core pipeline features a three-iteration dynamic retrieval loop, where two gpt-oss-120b-powered modules—an Iterative Query Generator and a Notes Builder—collaborate to refine queries and identify information gaps. Following this loop, an Answerability Classifier directs the query to one of three generation paths: Complete Answer, Partial Answer, or Clarification Request. The system employs a hybrid BM25 and dense retrieval approach, fused using Reciprocal Rank Fusion and enhanced by the Jina listwise reranker. The entire retrieval pipeline is built with DSPy and optimized via GEPA. The system achieved an nDCG@5 of 0.4502 (rank 17/38, Subtask A) and an HM of 0.3774 (rank 24/29, Subtask C). Post-hoc analysis revealed the Answerability Classifier as the primary bottleneck, flagging 75.5% of all responses as "IDK," including 69.8% of answerable questions, despite strong performance from other components.

Key takeaway

For Machine Learning Engineers building multi-turn RAG systems, consider implementing an iterative query refinement loop with a "Notes Builder" to address information gaps effectively. While this approach improves retrieval, you must carefully calibrate your answerability classifier. An overly conservative classifier, like the one flagging 75.5% of responses as "IDK" in this study, will severely limit your system's overall utility, even if retrieval and generation components perform well. Prioritize balancing false positives and negatives in your classifier design.

Key insights

Iterative query refinement and structured note-taking significantly enhance multi-turn RAG performance by addressing information gaps.

Principles

• Iterative query generation and note-building improve multi-turn RAG.
• Over-conservative answerability classifiers can severely bottleneck RAG systems.
• Hybrid retrieval fused with reranking enhances document relevance.

Method

A three-iteration dynamic retrieval loop uses gpt-oss-120b modules for query generation and note building, followed by an answerability classifier routing to specific generation paths, and hybrid BM25/dense retrieval refined by a listwise reranker.

In practice

• Integrate iterative query refinement using LLMs in RAG pipelines.
• Calibrate answerability classifiers to balance precision and recall.
• Employ hybrid retrieval (BM25, dense) with a listwise reranker.

Topics

• Multi-turn RAG
• Query Refinement
• Hybrid Retrieval
• Answerability Classifier
• DSPy
• SemEval-2026

Code references

• GuiiCorreia/SemEval-2026

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Advanced Retrieval Pipeline for RAG (HyDE, Hybrid Search, Reranking) | Build 100% Local Retrieval — Combining HyDE, hybrid search, and reranking significantly enhances RAG retrieval accuracy and relevance.
• An Efficient Approach for Answering Not Readily Attainable Questions for RAG-based Applications — This paper proposes an efficient approach for RAG systems to answer questions not readily found in retrieved data.
• When Iterative RAG Beats Ideal Evidence: A Diagnostic Study in Scientific Multi-hop Question Answering — Iterative RAG's staged retrieval and reasoning process outperforms static ideal evidence in complex multi-hop QA.
• RAG is dead, right?? — Kuba Rogut, Turbopuffer — Hybrid tool-rich retrieval and iterative agentic search are replacing simple RAG for advanced context understanding and performance gains.
• Scaling Retrieval-Augmented Reasoning with Parallel Search and Explicit Merging — MultiSearch uses parallel multi-query retrieval and explicit merging to enhance LLM reasoning and signal-to-noise ratio.
• I Built a RAG Evaluation Framework from Scratch. Here’s What Broke It. — Rigorous RAG evaluation reveals optimal configurations depend on query complexity and highlights the critical role of eval set quality.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.