Can LLMs Replace Survey Respondents?

2026-05-20 · Source: Towards Data Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Economic Analysis & Policy · Depth: Expert, medium

Summary

Large language models (LLMs) can replicate the median responses of household surveys, such as the Survey of Consumer Expectations (SCE), for inflation expectations to within a percentage point (e.g., 3% in 2020). However, LLMs suffer from "mode collapse," where 95% of simulated respondents cluster within a two-percentage-point window, failing to capture the wide dispersion of real human responses (e.g., -25% to +27% in 2020 SCE). This issue, likely caused by LLMs retrieving memorized statistics, is not resolved by standard prompting. Researchers applied unlearning methods, Gradient Ascent (GA) and Negative Preference Optimization (NPO), to Llama-3.1-8B-Instruct. GA achieved 97% tail accuracy against the SCE benchmark of 44% (responses >3pp from mode), and NPO achieved 98%, significantly improving distributional realism. In replicating a randomized controlled trial (RCT), Llama-GA successfully reproduced treatment effects and dispersion, unlike baseline or NPO models, suggesting unlearning enables more realistic synthetic agent behavior for complex simulations.

Key takeaway

For AI Scientists and Machine Learning Engineers developing synthetic survey agents, you must prioritize distributional accuracy over just matching averages. Standard LLMs exhibit "mode collapse," making them unsuitable for replicating heterogeneous populations or complex randomized controlled trials. You should explore unlearning methods like Gradient Ascent to remove memorized statistics. This enables models to generate more realistic response distributions and accurately simulate treatment effects. Always report second moments to validate your synthetic populations.

Key insights

LLMs can match survey means but fail to reproduce population dispersion due to "mode collapse" from memorized training data.

Principles

• LLMs default to narrow distributions.
• Memorized data overpowers prompt instructions.
• Distributional accuracy is crucial for simulations.

Method

Apply Gradient Ascent (GA) or Negative Preference Optimization (NPO) to unlearn specific data, maximizing prediction loss on forget sets while retaining general capabilities.

In practice

• Report second moments in synthetic surveys.
• Consider unlearning for diverse simulations.
• Evaluate models on belief updating.

Topics

• LLM Survey Simulation
• Mode Collapse
• Model Unlearning
• Gradient Ascent
• Inflation Expectations
• Randomized Controlled Trials

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Can Large Language Models Revolutionize Survey Research? Experiments with Disaster Preparedness Responses — Theory-constrained retrieval and demographic anchoring enable LLMs to improve survey data imputation and analysis, especially for vulnerable groups.
• Fake users generated by AI can't simulate humans — review of 182 research papers — LLMs fail to simulate human cognition and behavior due to their training on text, not diverse lived experience.
• UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs — LLMs consistently struggle to faithfully sample from target probability distributions, hindering their utility in stochastic simulations.
• The signal is the ceiling: Measurement limits of LLM-predicted experience ratings from open-ended survey text — Input text characteristics limit LLM prediction accuracy more than prompt or model choice.
• LLMs are stuck in a groupthink rut. This startup is trying to get them out. — Mainstream LLMs exhibit significant response homogeneity, which Springboards' Flint model addresses by selectively increasing output variety.
• Memory in LLMs: Weights and Activations - Jack Morris, Cornell — Current LLMs struggle with niche knowledge and context scaling; training knowledge into weights offers a path beyond RAG's limitations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.