Polarization by Default: Auditing Recommendation Bias in LLM-Based Content Curation

2026-04-17 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A controlled simulation study investigated content selection biases in Large Language Models (LLMs) used for content curation and ranking. Researchers mapped these biases across three major LLM providers (OpenAI, Anthropic, Google) using real social media datasets from Twitter/X, Bluesky, and Reddit. The study employed six prompting strategies: general, popular, engaging, informative, controversial, and neutral. Through 540,000 simulated top-10 selections from pools of 100 posts across 54 experimental conditions, the findings indicate that biases vary significantly in their structural nature and prompt sensitivity. Polarization was amplified across all configurations, toxicity handling showed an inversion between engagement- and information-focused prompts, and sentiment biases were predominantly negative. GPT-4o Mini exhibited the most consistent behavior, while Claude and Gemini showed high adaptivity in toxicity handling. Gemini also displayed the strongest negative sentiment preference. On Twitter/X, left-leaning authors were systematically over-represented despite forming a minority in the dataset, a pattern that largely persisted across prompts.

Key takeaway

For engineering teams deploying LLMs for content curation, you must actively audit and mitigate inherent biases. Be aware that polarization is amplified by default, and negative sentiment is common. Consider provider-specific trade-offs, such as GPT-4o Mini's consistency versus Claude and Gemini's toxicity adaptivity, and implement diverse prompting strategies to counter specific biases like political leaning over-representation.

Key insights

LLM content curation amplifies polarization and exhibits consistent negative sentiment, with varying provider-specific biases.

Principles

• LLM biases are both structural and prompt-sensitive.
• Polarization is a default outcome in LLM content curation.
• Sentiment bias is predominantly negative across LLM providers.

Method

The study used a controlled simulation with 540,000 top-10 selections from 100-post pools, testing three LLM providers and six prompting strategies on real social media data.

In practice

• GPT-4o Mini offers consistent LLM behavior.
• Claude and Gemini adapt well to toxicity handling.
• Prompt design influences toxicity and sentiment biases.

Topics

• LLM Content Curation
• Recommendation Bias
• Social Media Datasets
• Prompt Engineering
• Polarization Amplification

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, AI Ethicist, MLOps Engineer

Related on AIssential

• Polarization by Default: Auditing Recommendation Bias in LLM-Based Content Curation — LLM-based content curation consistently amplifies polarization and exhibits demographic biases, often resistant to prompt engineering.
• Are LLMs safe? — LLM training data curation, especially via quality filters, embeds implicit biases that shape model behavior and capabilities.
• AMEL: Accumulated Message Effects on LLM Judgments — Large language model judgments are significantly biased by the polarity of prior conversation history.
• Isolating LLM Lexical Bias: A Curation-Free Triangulated Metric for Preference-Stage Learning — A new metric quantifies LLM lexical bias from preference learning without manual curation.
• Recency Bias Is Architecture, Not Capability — Recency bias in LLMs is an architectural problem solvable by context window assembly, not model upgrades.
• State media control shapes LLM behaviour by influencing training data — State media control biases LLM behavior by influencing their training data, impacting information processing.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.