NCL at SemEval-2026 Task 8: Deterministic Small-LLM RAG with Relation Classification

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Software Development & Engineering · Depth: Advanced, short

Summary

NCL's system for SemEval-2026 Task 8B addresses multi-turn retrieval-augmented dialogue generation through a compact, reproducible RAG pipeline. The system integrates global and local question rewriting using LLM-based multi-turn relation control, followed by passage reranking with BGE-M3. It employs context-level answerability filtering via strict binary LLM judgments. For deterministic inference, NCL utilizes a small-LLM, Qwen2.5-1.5B-Instruct, augmented with a post-generation quality fallback mechanism including cleaning, a bad-answer gate, a stricter retry, and an "IDK" fallback. On the official test set, the system achieved a harmonic mean score of 0.5973 (RB${agg}$ 0.4993, RL$F$ 0.7235, RB${llm}$ 0.6105), placing 19th among 26 participating teams.

Key takeaway

For NLP Engineers developing multi-turn RAG systems, NCL's approach demonstrates that combining a small-LLM like Qwen2.5-1.5B-Instruct with a deterministic pipeline and robust quality fallbacks can yield competitive results. You should consider integrating strict binary LLM answerability filtering and multi-stage post-generation checks to improve reliability and control, even if your system doesn't rank top-tier. This strategy offers a reproducible path for resource-constrained deployments.

Key insights

A compact RAG pipeline using a small-LLM and strict filtering can achieve deterministic multi-turn dialogue generation.

Principles

• Multi-turn RAG benefits from explicit relation control.
• Binary LLM judgments enhance answerability filtering.
• Small-LLMs can be effective with robust fallbacks.

Method

The system employs LLM-based question rewriting, BGE-M3 for reranking, binary LLM filtering for answerability, and Qwen2.5-1.5B-Instruct with a multi-stage quality fallback for deterministic generation.

In practice

• Implement LLM-based question rewriting for context.
• Use BGE-M3 for efficient passage reranking.
• Integrate "yes"/"no" LLM gates for answerability.

Topics

• Retrieval-Augmented Generation
• Small Language Models (Small-LLMs)
• SemEval-2026 Task 8B
• Multi-turn Dialogue Systems
• Relation Classification
• Qwen2.5-1.5B-Instruct
• BGE-M3

Best for: Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• How to Teach the LLM to Think With Your Data — Feeding RAG-retrieved context into an LLM enables reasoning, reducing hallucinations and producing more natural, coherent answers.
• Announcing the ICLR 2026 Outstanding Papers — Transformer efficiency stems from succinct concept encoding, while LLMs struggle with multi-turn conversations due to training data mismatches.
• Don't Ask the LLM to Track Freshness: A Deterministic Recipe for Memory Conflict Resolution — Deterministic aggregation, not LLM judgment, is key for resolving memory conflicts in evolving fact systems.
• Combining Retrieval-Augmented Text Generation with LLMs for Reading Content Recommendations — RAG significantly enhances LLM-generated personalized reading content, improving relevance and groundedness.
• Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement — Decomposing LLM evaluation into binary questions provides interpretable, multi-dimensional scores and enables self-improvement.
• Building HITL Feedback RAG: Embeddings, Retrieval, and Reranking — HITL Feedback RAG improves LLM accuracy by dynamically injecting human-curated corrections via a robust retrieval and reranking pipeline.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.