One More Turn, Less Regret: A Regret-Based Multi-Turn Benchmark for LLMs' Clarification Policies

2025-11-17 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Conversational AI · Depth: Expert, extended

Summary

RegretBench is a new multi-turn benchmark designed to evaluate Large Language Models' (LLMs) clarification policies, moving beyond isolated question quality to assess sequential decision-making under ambiguous user requests. It features a hidden-intent formulation of ambiguity, supports free-form interaction grounded in semantic-state tracking, and introduces a regret-based objective. This objective measures value loss relative to a reference clarification policy, combining final intent correctness, interaction cost, and penalties for ineffective clarification. Experiments across open-domain QA and product recommendation scenarios demonstrate that final success rate alone is insufficient, as models like K2-Think V2 and Llama 4 Scout can achieve high accuracy but suffer from excessive turns and costs, while GPT 4.1 and DeepSeek V4 Flash show better efficiency.

Key takeaway

For ML engineers developing conversational LLMs, relying solely on final answer accuracy to evaluate clarification policies is misleading. You should adopt policy-level evaluation frameworks like RegretBench to assess whole-dialogue performance, considering interaction cost, efficiency, and robustness to user behavior. This approach reveals whether your models clarify usefully and efficiently, ensuring they ask the right questions at the right time and stop appropriately, rather than just eventually getting the correct answer.

Key insights

Clarification for LLMs is a sequential decision problem requiring efficient, grounded interaction to resolve hidden user intent.

Principles

• Final success rate alone is insufficient for evaluating clarification.
• Effective clarification demands asking the right question at the right time.
• Models must stop clarifying once user intent is clear.

Method

RegretBench treats the assistant as a dialogue policy in a partially observed environment, mapping free-form natural language to semantic actions for intent tracking, user simulation, and whole-dialogue reward computation.

In practice

• Evaluate LLM clarification policies using whole-dialogue reward and regret metrics.
• Apply prompt-based interventions to shift efficiency-accuracy trade-offs.
• Test model robustness across diverse user personas (cooperative, vague, contradictory).

Topics

• LLM Evaluation
• Clarification Policies
• Multi-turn Dialogue
• RegretBench
• Ambiguity Resolution
• Conversational AI
• User Simulation

Code references

• ngocminhta/RegretBench

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• One More Turn, Less Regret: A Regret-Based Multi-Turn Benchmark for LLMs' Clarification Policies — LLM clarification is a sequential decision problem requiring optimal timing and stopping, not just isolated question quality.
• Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment — LLM-based agents struggle with NFR assessment accuracy despite user agreement, with dialogue length and proactivity impacting satisfaction.
• When the Database Fails: Prompting LLM Dialogue Agents for Safe Recovery in Task-Oriented Dialogue — Prompting with structured database status significantly reduces LLM hallucination in task-oriented dialogue without retraining.
• AttuneBench: A Conversation-Based Benchmark for LLM Emotional Intelligence — LLM emotional intelligence is a multi-faceted capability, requiring distinct evaluation metrics beyond simple emotion recognition.
• Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement — Decomposing LLM evaluation into binary questions provides interpretable, multi-dimensional scores and enables self-improvement.
• Talking to a Know-It-All GPT or a Second-Guesser Claude? How Repair reveals unreliable Multi-Turn Behavior in LLMs — LLM multi-turn conversational repair behavior is unreliable and highly model-specific, challenging consistent user interaction.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.