Rushes: A Human Preference Dataset for Pluralistic Alignment

2025-09-22 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

Microsoft Research introduces Rushes, a new human preference dataset and benchmark designed for studying engagement in interactive narrative environments. Collected from 8,167 unique Xbox Insiders across six AI-generated branching narrative games, Rushes contains 44,226 decision events, logging user choices, candidate sets, and evolving narrative context. This dataset captures sequential, personalized engagement, revealing structured, non-random user choices with a mean entropy of 1.04 nats, significantly lower than a uniform four-choice baseline of 1.39 nats. Benchmarking against Rushes exposes an "Engagement Gap": state-of-the-art LLMs, including GPT-5, achieve only 34.2% accuracy with user history, failing to outperform a simple popularity baseline (36.4%) or an SVD-based matrix factorization model (37.7%) in predicting event-level choices. This suggests current LLM alignment methods struggle with heterogeneous, context-dependent engagement signals.

Key takeaway

For AI Scientists and Machine Learning Engineers developing interactive generative systems, recognize that current LLMs like GPT-5 struggle with personalized engagement. Your models may default to majority preferences, missing individual user trajectories. You should explore Rushes to benchmark and develop new alignment techniques that capture subjective, context-dependent preferences, moving beyond population-level objectives. Consider architectural innovations to model "fun" and "interestingness" effectively.

Key insights

Current LLMs struggle to predict personalized engagement in interactive narratives, underperforming simpler baselines.

Principles

• Engagement alignment requires personalized, sequential preference data.
• Population-level objectives may suppress minority viewpoints.
• Revealed preferences from actions are more reliable than surveys.

Method

Rushes collects revealed human engagement preferences from AI-generated branching narratives, logging user choices, candidate sets, and evolving context to create time-ordered trajectories.

In practice

• Use Rushes to benchmark personalized alignment models.
• Explore architectural innovations for subjective engagement.
• Analyze context-dependent preference transfer learning.

Topics

• Human Preference Datasets
• Pluralistic Alignment
• Interactive Narratives
• Large Language Models
• Engagement Modeling
• Reinforcement Learning from Human Feedback
• GPT-5

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Rushes: A Human Preference Dataset for Pluralistic Alignment — Leading LLMs struggle with personalized engagement in interactive narratives, revealing limitations of current population-level alignment methods.
• Large Language Models Should Learn Personalized Rather Than Aggregated Human Preferences — LLMs should learn personalized human preferences, not aggregated ones, to avoid optimizing for a hypothetical "average user" and better respect individual autonomy.
• PolyAlign: Conditional Human-Distribution Alignment — PolyAlign aligns language models to context-specific human response distributions, overcoming limitations of global alignment for improved naturalness and faithfulness.
• Response Time Enhances Alignment with Heterogeneous Preferences — Response time data can restore identifiability of population-average preferences in LLM alignment, even with heterogeneous, anonymous labelers.
• What Do People Actually Want From AI? Mapping Preference Plurality — Human preferences for AI are pluralistic and contextual, challenging current singular alignment methods.
• Align AI to Dynamic Human-AI Workflows — AI alignment must evolve from static preference emulation to dynamic, interactive human-AI co-evolution.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.