AI can predict how you’ll respond to a survey. But that’s not the same as understanding you

2026-07-09 · Source: Artificial intelligence (AI) – The Conversation · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Social Sciences & Behavioral Studies · Depth: Intermediate, short

Summary

A new study led by Harvard psychology researcher Ashwini Ashokkumar, published in Nature, reveals that large language models (LLMs) like GPT-4 can surprisingly well predict outcomes of many social science experiments. The research compared GPT-4's estimations against results from 70 real US experiments involving nearly 120,000 participants, finding a strong correlation in distinguishing intervention effectiveness. While LLMs capture meaningful patterns in text-based survey experiments, the study cautions that prediction does not equate to understanding; GPT-4 systematically overestimated effects by approximately double. These "synthetic respondents" are not direct substitutes for real people. LLMs offer value for pilot studies, helping refine interventions and estimate effect sizes, especially when combined with human forecasts. However, risks include undermining public trust if "silicon sampling" is mistaken for genuine public opinion, reproducing dominant patterns, and potential misuse for optimizing harmful persuasion. Reproducibility is also a concern with proprietary LLMs.

Key takeaway

For social scientists and market researchers designing experiments or forecasting public opinion, you should integrate LLM predictions into pilot studies to refine interventions and explore scenarios efficiently. However, critically validate these "silicon samples" against real human data, as LLMs predict patterns without true understanding and can reproduce biases. Avoid mistaking model-generated proxies for genuine public sentiment, and implement safeguards against optimizing harmful persuasion.

Key insights

LLMs predict social experiment outcomes but lack understanding, making them valuable tools for pilots, not human substitutes.

Principles

• Prediction does not equal understanding.
• Synthetic samples are not real people.
• Combine AI and human forecasts for accuracy.

Method

Researchers provided GPT-4 with hypothetical respondent profiles, experimental messages, and survey questions to predict responses, comparing these to actual experiment results.

In practice

• Conduct LLM-assisted pilot studies to refine interventions.
• Simulate demographic responses to various messages or policies.
• Integrate LLM predictions with human expert forecasts.

Topics

• Large Language Models
• Social Science Research
• GPT-4
• Synthetic Data
• Research Methodology
• AI Ethics

Best for: AI Scientist, Research Scientist, AI Ethicist

Related on AIssential

• Who Speaks for Whom? LLM-Generated Survey Data as a Proxy for Public Opinion — LLM-generated survey data lacks human nuance and reliability, showing biases and inconsistent reasoning.
• Evaluating LLMs as Human Surrogates in Controlled Experiments — LLMs can reproduce directional behavioral effects but often miscalibrate effect magnitudes compared to human data.
• Can LLMs Replace Survey Respondents? — LLMs can match survey means but fail to reproduce population dispersion due to "mode collapse" from memorized training data.
• AI is replacing humans in responding to some surveys – but simulated opinions are not the same as public opinion — Simulating public opinion with AI models is not equivalent to measuring actual public opinion due to inherent biases and lack of real-world validation.
• Fake users generated by AI can't simulate humans — review of 182 research papers — LLMs fail to simulate human cognition and behavior due to their training on text, not diverse lived experience.
• When Do LLMs Generate Realistic Social Networks? A Multi-Dimensional Study of Culture, Language, Scale, and Method — LLM-generated social networks are significantly influenced by prompt design, culture, language, and model scale.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial intelligence (AI) – The Conversation.