Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBench

2026-05-19 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Marketing, Branding & Advertising · Depth: Expert, extended

Summary

Researchers have introduced ConsumerSimBench, a new benchmark designed to evaluate how accurately Large Language Models (LLMs) can reconstruct real consumer reaction patterns in public discourse. Built from 1,553 Chinese social-media topics and 23,122 atomic, rule-audited criteria across four reaction families (sentiment flashpoints, emotion keywords, positive aspects, and negative aspects), the benchmark moves beyond holistic preference scoring. Instead, it uses auditable yes-no decisions for concrete reaction points, achieving 92.1% three-judge agreement and 98.4% agreement with human-majority labels. Testing 13 frontier LLMs, the study found that the strongest model, Gemini-3.1-Pro, covered only 47.8% of real reaction criteria. Other models like GPT-5.2 and Claude-4.6 performed significantly worse, highlighting a substantial gap between technical benchmark performance and socially grounded consumer intuition. The research indicates that LLMs struggle with anticipating specific social triggers and criticism vectors, even when generating fluent, emotionally varied comments.

Key takeaway

For AI Product Managers developing LLM-based consumer simulation tools, recognize that current frontier models are not yet reliable for forecasting specific public discourse reactions. Your systems may generate fluent, emotionally plausible comments but still miss critical sentiment flashpoints and negative aspects that drive real-world engagement or crises. Prioritize integrating iterative refinement pipelines and focus on benchmarking against concrete, auditable reaction criteria rather than relying on holistic preference scores to ensure your models can anticipate what consumers will actually care about.

Key insights

LLMs struggle to reconstruct specific, socially grounded consumer reactions in public discourse, despite strong technical performance.

Principles

• Authenticity over generic plausibility is crucial for consumer simulation.
• Auditable scoring of concrete reaction points is more reliable than holistic LLM-as-Judge.
• Socially charged anchors are harder for LLMs to predict than generic sentiment.

Method

ConsumerSimBench uses 1,553 real social-media topics and 23,122 atomic criteria across four reaction families. LLMs generate comments, which are then evaluated via binary yes-no decisions against these criteria by an LLM judge.

In practice

• Use multi-agent pipelines for iterative refinement in consumer simulation.
• Focus on specific cultural anchors and first-person consumer voice in prompts.
• Prioritize anticipating criticism vectors and emotional flashpoints for marketing.

Topics

• ConsumerSimBench
• LLM Consumer Simulation
• Crowd Reaction Forecasting
• Social Media Discourse
• Sentiment Flashpoints

Best for: Research Scientist, AI Product Manager, AI Scientist, Machine Learning Engineer, Director of AI/ML

Related on AIssential

• SocialPersona: Benchmarking Personalized Profiling and Response with Multimodal Social-Media Context — SocialPersona benchmarks MLLMs' ability to infer and act on user preferences from multimodal social media data for personalized dialogue.
• Analysing Differences in Persuasive Language in LLM-Generated Text: Uncovering Stereotypical Gender Patterns — LLMs exhibit pervasive gender-stereotypical persuasive language biases across models and languages, reflecting societal norms.
• GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations — LLM agent memory systems perform poorly in multi-party conversations, often outperformed by simple baselines.
• What sentiment analysis can't see: Measuring whether customers were helped, and what went wrong, across 70,000 support conversations — LLM-based analysis of support conversations surpasses sentiment analysis in measuring customer satisfaction and identifying underlying problems.
• I built a public voting benchmark where models have to make memes out of daily news — Memebench evaluates LLM humor and creativity by generating memes from daily news for public A/B voting.
• A Model of Multi-turn Human Persuadability Using Probabilistic Belief Tracing — PERSUASIONTRACE offers a framework and Bayesian target for analyzing multi-turn human-LLM persuasion dynamics with process fidelity.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.