What Do People Actually Want From AI? Mapping Preference Plurality

2026-06-04 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Social Sciences & Behavioral Studies · Depth: Expert, quick

Summary

Large Language Models (LLMs) are often fine-tuned using Reinforcement Learning from Human Feedback (RLHF) to align with user preferences, but this method has significant limitations. An analysis of 1,500 open-ended responses from the PRISM dataset across 75 countries reveals that people's preferences are highly diverse; most values are requested by fewer than a quarter of respondents, with truthfulness being the sole exception at 49%. Crucially, the term "truthfulness" itself hides divergent meanings, encompassing requests for sourced claims, expert opinions, or even unpopular views. Furthermore, capabilities like human-like model behavior and features such as AI guardrails are controversial. The study also found that users make contextual distinctions (e.g., "by default" versus "if requested") that binary comparisons fail to capture, exposing fundamental problems in current alignment practices and explaining persistent hallucination rates.

Key takeaway

For AI Scientists and Ethicists developing alignment strategies, you must move beyond aggregated, binary preference models. Your current RLHF methods likely flatten diverse user values and misinterpret terms like "truthfulness." You should design systems that accommodate preference plurality and contextual distinctions (default vs. requested) to genuinely address user needs and reduce issues like persistent hallucinations.

Key insights

People's AI preferences are diverse, context-dependent, and often contradictory, challenging current single-model alignment methods.

Principles

• Aggregating preferences flattens diverse values.
• Same words hide divergent meanings.
• Contextual distinctions are crucial for preferences.

In practice

• Avoid binary comparisons for complex preferences.
• Probe deeper into stated user values.
• Recognize controversial AI features.

Topics

• Large Language Models
• Reinforcement Learning from Human Feedback
• AI Alignment
• User Preferences
• Epistemic Violence
• PRISM dataset

Best for: Research Scientist, AI Scientist, AI Ethicist

Related on AIssential

• What Do People Actually Want From AI? Mapping Preference Plurality — Human preferences for AI are pluralistic and contextual, challenging current singular alignment methods.
• In-Context Reward Adaptation for Robust Preference Modeling — In-Context Reward Adaptation uses transformers and human response time to dynamically model diverse, unseen human preferences for robust RLHF.
• Align Large Language Model with Human Preference — RLHF aligns LLMs with human values for helpful, honest, and harmless text generation by leveraging human feedback.
• Young people's perceptions and recommendations for conversational generative artificial intelligence in youth mental health — Young people prioritize transparency, human connection, and personalized, safe integration of genAI chatbots in mental health.
• Preference tuning explained — Preference tuning teaches models to generate humanlike responses by learning from preferred answer pairs.
• Large Language Models Should Learn Personalized Rather Than Aggregated Human Preferences — LLMs should learn personalized human preferences, not aggregated ones, to avoid optimizing for a hypothetical "average user" and better respect individual autonomy.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.