Who Speaks for Whom? LLM-Generated Survey Data as a Proxy for Public Opinion

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Science & Research — Social Sciences & Behavioral Studies, Research Methodology & Innovation, Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

The 2026 paper "Who Speaks for Whom? LLM-Generated Survey Data as a Proxy for Public Opinion" by Venkatakrishnan, Brodbeck, and Young investigates using Large Language Models (LLMs) for synthetic survey data. This method aims to address rising costs and declining response rates in social science research. LLMs role-play artificial personas with specific demographic traits. The authors present a procedure to evaluate synthetic data quality. This involves measuring Intra Class Correlation (ICC), Earth Mover Distance (EMD), Variance, Hedging, and demographic drivers of LLM output. Findings show LLMs produce plausible aggregate results, but synthetic data lacks human depth and nuance. LLM reasoning, even for ten-point scale answers, often exhibits hedging inconsistent with the response. Data distortion was not uniform; some demographic groups experienced more extreme effects. This suggests current LLM technology for synthetic survey data is not yet mature for public opinion research.

Key takeaway

For social science researchers considering Large Language Models (LLMs) for public opinion surveys, you should exercise significant caution. Current LLM-generated data lacks the depth and nuance of human responses, and its reasoning can be inconsistent. Relying on this data risks misrepresenting public opinion, especially for specific demographic groups where distortions are more extreme. Prioritize human-centric data collection until LLM technology demonstrably overcomes these critical limitations in accuracy and reliability.

Key insights

LLM-generated survey data lacks human nuance and reliability, showing biases and inconsistent reasoning.

Principles

• Synthetic data may appear plausible in aggregate.
• LLM training can bias results towards WEIRD cultures.
• Hedging in LLM reasoning can contradict definitive answers.

Method

Evaluate synthetic data quality by measuring Intra Class Correlation (ICC), Earth Mover Distance (EMD), Variance, Hedging, and demographic drivers of LLM output.

In practice

• Use ICC to assess inter-rater reliability.
• Apply EMD for distribution similarity.
• Analyze hedging to detect reasoning inconsistencies.

Topics

• Large Language Models
• Synthetic Data Generation
• Public Opinion Research
• Survey Data Quality
• Demographic Bias
• Computational Social Science

Best for: Research Scientist, AI Scientist, Data Scientist

Related on AIssential

• The 1.6 Billion People Large Language Models Still Can’t Understand — Large language models fail profoundly in many non-English languages, going beyond mere performance degradation to confident incorrectness.
• The threat of analytic flexibility in using large language models to simulate human data — Analytic choices in LLM-generated synthetic data significantly alter its fidelity to human responses.
• Can Large Language Models Revolutionize Survey Research? Experiments with Disaster Preparedness Responses — Theory-constrained retrieval and demographic anchoring enable LLMs to improve survey data imputation and analysis, especially for vulnerable groups.
• Study: AI chatbots provide less-accurate information to vulnerable users — AI chatbots underperform for vulnerable users, providing less accurate information and exhibiting biased behaviors.
• LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation — LLMs can generate synthetic data to validate NLP evaluation metrics, replacing costly human annotations.
• Can LLMs Replace Survey Respondents? — LLMs can match survey means but fail to reproduce population dispersion due to "mode collapse" from memorized training data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.