UXBench: Benchmarking User Experience in AI Assistants

2026-07-15 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, extended

Summary

UXBench is introduced as the first user-centric benchmark designed to evaluate user experience (UX) in AI assistants, moving beyond general model capabilities. Developed by Hong Kong Polytechnic University Yuanbao Team and Tencent, it comprises 7,400 test instances derived from over 70,000 real interaction logs of a mainstream Chinese AI assistant, spanning 8 scenarios and 83 domains. The benchmark features three tasks: UX Judge, UX Eval, and UX Recovery, assessing user feedback prediction, response generation quality, and failure recovery. Experiments across 26 frontier language models revealed a systematic positive bias in LLM judges, a weak scaling trend between model capability and UX, and the critical role of response strategies. A specialized generative reward model, trained on 8,547 positive and 8,559 negative in-the-wild instances, achieved superior accuracy in predicting user feedback, demonstrating UX judgment as a learnable capability.

Key takeaway

For AI Product Managers or ML Engineers developing AI assistants, you should prioritize user-centric evaluation beyond traditional capability benchmarks. Focus on training specialized reward models with real user feedback to accurately predict satisfaction and guide response generation. Your development efforts should include tailored UX optimization and refining conversational strategies, as general model scaling alone offers only modest UX gains.

Key insights

Real user feedback is essential for AI assistant UX evaluation, revealing systematic LLM biases and weak scaling trends.

Principles

• LLMs show strong positive bias in UX judgment.
• UX improvements weakly scale with model capability.
• Response strategies are key for user satisfaction.

Method

UXBench defines three tasks (Judge, Eval, Recovery) using 7,400 test cases from 70K+ real interaction logs. It derives ground-truth UX labels from explicit and implicit user feedback, including a "gap_ratio" for skimming.

In practice

• Train reward models with real user feedback.
• Prioritize natural, conversational response styles.
• Model user profiles for improved UX judgment.

Topics

• UXBench
• AI Assistant UX
• Reward Modeling
• LLM Evaluation Bias
• Dialogue Generation
• User Feedback Signals

Code references

• tatsu-lab/alpaca_eval

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Product Manager

Related on AIssential

• UXBench: Measuring the Actionability of LLM-Generated UX Critiques — LLM UX critique actionability varies significantly, requiring interaction-grounded evaluation beyond static analysis.
• Personalized Turn-Level User Conversation Satisfaction Benchmark — Personalized user satisfaction in AI conversations demands evaluation methods that integrate user memory and turn-level context, moving beyond generic response quality.
• Introducing BenchBench — BenchBench reveals a critical divergence between LLM "Creator" and "Solver" capabilities, with GPT 5.2 excelling at benchmark generation.
• What Do Models Still Suck At? - Peter Gostev, Arena.ai, BullshitBench — LLMs still struggle with identifying nonsense and consistently meeting user expectations in complex, real-world tasks.
• I benchmarked the NEW Sonnet 5. The results shocked me. — Effective LLM benchmarking requires balancing automated metrics with nuanced human qualitative judgment to capture true model utility.
• Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach — MLLMs can diagnose complex UI/UX issues by combining task-adaptive rewards and reasoning efficiency penalties.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.