Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging

2026-07-27 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Natural Language Processing · Depth: Expert, extended

Summary

EYT-Bench is a new human-centered benchmark for evaluating large language models (LLMs) in multi-turn dialogue, addressing limitations of single-turn metrics. Developed by SenseTime Research and universities, it features a three-party decoupled design: a persona-grounded user simulator, a target model with decoupled intent perception and response generation, and an independent LLM judge. Personas are sampled from human-curated corpora, Nemotron-Personas-USA (NVIDIA, 2025) and PersonaMem-v2 (Jiang et al., 2025). The benchmark introduces novel trajectory-level metrics like embedding-based "intent-drift" and "final-intent completion rate" (FICR). A $17$-target $ imes$ $200$-dialogue evaluation revealed that subjective scores for closed- and open-source models are statistically indistinguishable, yet objective intent-tracking varies by up to $9\times$. Reasoning significantly improves objective tracking on long-context personas (e.g., $+0.47$–\$0.50$ latent-intent accuracy on Gemma-4), and persona format critically influences trajectory spread, with PersonaMem-v2 proving more discriminative (FICR spreads \$0.53\to 0.88$).

Key takeaway

For AI Scientists and Machine Learning Engineers evaluating LLMs for complex conversational applications, you should prioritize benchmarks that employ decoupled evaluation and trajectory-level metrics. Subjective Likert scores alone are insufficient for differentiating frontier models. Focus on objective intent tracking and final-intent completion rates, especially with long-context, free-text personas like PersonaMem-v2, to accurately assess a model's true conversational capabilities and goal convergence.

Key insights

Decoupled design and trajectory metrics are essential for robust multi-turn LLM evaluation beyond subjective scores.

Principles

• Decouple simulator, target, and judge to prevent self-preference bias.
• Human-curated personas reduce LLM-induced persona bias.
• Persona format significantly impacts benchmark discriminative power.

Method

EYT-Bench employs a three-party decoupled protocol with a persona-grounded user simulator, a target model separating intent perception from response generation, and an independent LLM judge. It introduces "intent-drift" and "final-intent completion rate" (FICR) metrics.

In practice

• Prioritize PersonaMem-v2 for discriminative trajectory evaluation.
• Enable reasoning for long-context persona tasks to boost objective tracking.
• Focus on objective trajectory metrics for frontier LLM comparison.

Topics

• Multi-turn Dialogue Evaluation
• Large Language Models
• Benchmark Design
• Persona-Grounded Dialogue
• LLM-as-a-Judge
• Intent Tracking

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Rethinking Evaluation in Retrieval-Augmented Personalized Dialogue: A Cognitive and Linguistic Perspective — Surface-level metrics fail to evaluate personalized dialogue; human and LLM judgments align on deeper cognitive and linguistic qualities.
• SocialPersona: Benchmarking Personalized Profiling and Response with Multimodal Social-Media Context — SocialPersona benchmarks MLLMs' ability to infer and act on user preferences from multimodal social media data for personalized dialogue.
• Low-Latency Turn-Taking via Context-Aware Preface Generation in a Real-World Dialogue Robot — A two-stage framework reduces LLM dialogue latency by decoupling preface generation from final speech recognition.
• GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations — LLM agent memory systems perform poorly in multi-party conversations, often outperformed by simple baselines.
• Evaluating Multi-turn Human-AI Interaction — Current NLP evaluation overlooks critical human-AI interaction behaviors like consistency and refinement.
• Conversational Grounding in Large Language Models: Evaluation Methods, Challenges and Future Directions — Evaluating conversational grounding in LLMs requires both explicit and implicit methods, facing significant methodological challenges.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.