Large Language Models Should Learn Personalized Rather Than Aggregated Human Preferences

2026-05-30 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

A position paper published on 2026-05-30 argues that current large language model (LLM) alignment methods, which aggregate diverse human preferences into a single reward signal, create an "average user" model that fails to serve real individuals effectively. This aggregation masks critical information regarding preference diversity, individual values, and contextual dependencies, a limitation grounded in social choice theory and evident across demographic groups. The paper advocates for LLMs to learn personalized, individual preferences instead. It analyzes the rich structure of human preferences, surveys technical approaches to personalization, and systematically addresses counterarguments concerning scalability, shared standards, and manipulation risk. While acknowledging genuine safety challenges like filter bubbles and psychological manipulation, the authors propose these are manageable through bounded personalization frameworks that maintain universal safety constraints while accommodating legitimate individual variation. The paper concludes with a concrete research and policy agenda.

Key takeaway

For AI Scientists and Policy Makers developing large language models, current approaches aggregating human preferences are fundamentally flawed and create an "average user" model. You should prioritize research into personalized preference learning, ensuring models respect individual autonomy while adhering to universal safety constraints. Implement "bounded personalization frameworks" to mitigate risks like filter bubbles and psychological manipulation. This shift requires a concerted research and policy agenda to build truly preference-aware and safe AI systems.

Key insights

LLMs should learn personalized human preferences, not aggregated ones, to avoid optimizing for a hypothetical "average user" and better respect individual autonomy.

Principles

• Aggregating preferences masks critical diversity.
• Bounded personalization manages safety challenges.
• Social choice theory limits preference aggregation.

Topics

• Large Language Models
• Human Preferences
• AI Alignment
• Personalization
• Social Choice Theory
• AI Safety
• Bounded Personalization

Best for: Research Scientist, AI Scientist, AI Ethicist, Policy Maker

Related on AIssential

• What Do People Actually Want From AI? Mapping Preference Plurality — Human preferences for AI are pluralistic and contextual, challenging current singular alignment methods.
• Language Models Exhibit Inconsistent Biases Towards Algorithmic Agents and Human Experts — LLMs exhibit inconsistent biases, preferring human experts in stated trust but algorithms in revealed preference tasks.
• Toward AI Systems That Understand Self and Others: A Multi-Phase Inference Framework for Human Cognitive Diversity and World-Model Alignment — AI systems can foster understanding by processing diverse world models, not forcing agreement.
• ‘Probably’ doesn’t mean the same thing to your AI as it does to you — AI chatbots misinterpret words of estimative probability compared to humans, posing risks in high-stakes applications.
• CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectives — LLMs struggle with nuanced human value judgments, including ambivalence, discomfort, and dynamic value shifts in high-stakes dilemmas.
• Personalized Benchmarking: Evaluating LLMs by Individual Preferences — Individual LLM preferences diverge significantly from aggregate benchmarks, driven by unique user query topics and writing styles.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.