Can Large Language Models Revolutionize Survey Research? Experiments with Disaster Preparedness Responses

2026-05-21 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Social Sciences & Behavioral Studies · Depth: Expert, extended

Summary

A study evaluates a five-stage framework for integrating large language models (LLMs) into survey research, using the 2024 Hurricane Milton preparedness survey of 946 Florida residents as a testbed. The framework covers questionnaire design, sample selection, pilot testing, missing-data imputation, and post-collection analysis. Researchers introduced a Protection Motivation Theory (PMT)-constrained co-occurrence knowledge graph to develop seven LLM configurations. Their proposed Anchored Marginal Theory-Informed LLM (A-TLM) significantly outperformed three classical imputation baselines (IPW/MI, MICE+PMM, missForest) on root-mean-square error (RMSE) under severe block-wise missing-not-at-random conditions (S4 RMSE 1.439 vs. 1.496 for the next-best method). A-TLM also achieved near-zero overall signed bias (-0.121), contrasting with the random-forest imputer's larger absolute bias (-0.631). The study highlights that near-zero aggregate bias can mask substantial opposing subgroup errors, particularly for compound-vulnerable respondents, and proposes subgroup-stratified bias auditing. A retrieval-constrained knowledge-graph chatbot demonstrated architectural management of hallucination risk.

Key takeaway

For Data Scientists and Research Scientists working with survey data, you should integrate theory-constrained LLMs into your imputation workflows. This approach, particularly the A-TLM, significantly reduces bias for compound-vulnerable subgroups under severe missingness. You must adopt subgroup-stratified bias reporting as a standard practice for policy-relevant applications, avoiding masked critical errors. Consider using LLM-augmented tools for early design stages and data analysis, ensuring architectural grounding to manage hallucination risks.

Key insights

Theory-constrained retrieval and demographic anchoring enable LLMs to improve survey data imputation and analysis, especially for vulnerable groups.

Principles

• Organize retrieval around theoretical structures.
• Integrate all evidence in a single model call.
• Audit subgroup-stratified bias.

Method

A five-stage LLM integration framework covers questionnaire design, sample selection, pilot testing, missing-data imputation, and post-collection analysis, using a PMT-constrained knowledge graph.

In practice

• Use LLMs for instrument audit and gap identification.
• Forecast sample under-representation pre-fielding.
• Ground chatbot answers in knowledge graphs.

Topics

• Large Language Models
• Survey Research
• Missing Data Imputation
• Retrieval-Augmented Generation
• Disaster Preparedness
• Protection Motivation Theory
• Bias Auditing

Best for: AI Scientist, Research Scientist, Data Scientist

Related on AIssential

• In-Context Learning for the Imputation of Public Opinion Data with Large Language Models — In-context learning with LLMs significantly improves missing public opinion data imputation over traditional statistical methods.
• Can LLMs Replace Survey Respondents? — LLMs can match survey means but fail to reproduce population dispersion due to "mode collapse" from memorized training data.
• Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Research — LLMs offer transformative potential for low-resource language research in humanities, despite significant data and ethical challenges.
• Isolating LLM Lexical Bias: A Curation-Free Triangulated Metric for Preference-Stage Learning — A new metric quantifies LLM lexical bias from preference learning without manual curation.
• LLM Watch Weekly: When Scale Isn't Enough — Scaling alone cannot overcome data biases or fundamental reasoning gaps in LLMs and VLMs.
• Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most — More capable LLMs make worse forecasts for superlinear growth and tail risk, especially in the upper tail.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.