Prompt Perturbations Reveal Human-Like Biases in Large Language Model Survey Responses

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Social Sciences & Behavioral Studies, Research Methodology & Innovation · Depth: Expert, quick

Summary

A study investigated the reliability and response robustness of 18 Large Language Models (LLMs) when used as proxies for human subjects in social science surveys. Researchers tested these LLMs on questions from the World Values Survey (WVS), applying a comprehensive set of ten perturbations to both question phrasing and answer option structure. This extensive testing generated over 334,800 simulated survey interviews. The findings reveal that LLMs are vulnerable to these perturbations, with almost all tested models exhibiting a consistent recency bias, disproportionately favoring the last-presented answer option. While larger models generally demonstrated greater robustness, all models remained sensitive to semantic variations, such as paraphrasing, and to combined perturbations. This research underscores the critical need for careful prompt design and thorough robustness testing when employing LLMs to generate synthetic survey data.

Key takeaway

For research scientists or NLP engineers using LLMs to generate synthetic survey data, you must prioritize rigorous prompt design and extensive robustness testing. Be aware that LLMs consistently exhibit recency bias, favoring the last answer option, and are sensitive to subtle semantic changes. You should systematically vary question phrasing and answer option order to mitigate these human-like biases and ensure the reliability of your simulated survey results.

Key insights

LLMs used in surveys exhibit human-like biases, especially recency bias, necessitating robust prompt design and testing.

Principles

• LLMs show consistent recency bias in surveys.
• Larger LLMs offer more response robustness.
• Semantic variations impact LLM survey responses.

Method

18 LLMs were tested on World Values Survey questions using ten perturbations to phrasing and answer options, generating over 334,800 simulated interviews.

In practice

• Design prompts to mitigate recency bias.
• Conduct robustness testing for LLM surveys.
• Vary answer option order in LLM surveys.

Topics

• Large Language Models
• Survey Research
• Response Bias
• Prompt Engineering
• Robustness Testing
• Synthetic Data

Best for: AI Engineer, Machine Learning Engineer, AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• LLMs Can Better Capture Human Judgments--With the Right Prompts — Asking better questions to LLMs through specific prompting strategies improves their ability to capture human judgments and response distributions.
• Can Large Language Models Revolutionize Survey Research? Experiments with Disaster Preparedness Responses — Theory-constrained retrieval and demographic anchoring enable LLMs to improve survey data imputation and analysis, especially for vulnerable groups.
• Cognitive Effects and Biases in Large Language Models — This tutorial clarifies cognitive effects and biases in LLMs by bridging psychology and NLP.
• Recency Bias Is Architecture, Not Capability — Recency bias in LLMs is an architectural problem solvable by context window assembly, not model upgrades.
• The signal is the ceiling: Measurement limits of LLM-predicted experience ratings from open-ended survey text — Input text characteristics limit LLM prediction accuracy more than prompt or model choice.
• When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variations — Healthcare LLMs are not intrinsically safe, exhibiting high sensitivity to prompt variations that can lead to harmful clinical outputs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.