Simulating Social Attitudes with LLMs: Accuracy, Demographic Effects, and Refusal Behavior in the Sensitive Domain of Suicide Prevention

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Social Sciences & Behavioral Studies · Depth: Expert, quick

Summary

A study evaluated the capacity of large language models (LLMs) to accurately simulate public attitudes regarding suicide prevention policies. Utilizing 32 questions from seven U.S. national surveys conducted between 2023 and 2025, researchers tested models including GPT-5 Nano, DeepSeek V3.2, Meta Llama 3.1 8B, and Mistral Small 24B. The evaluation systematically varied demographic conditioning (race/ethnicity, gender, age, education, income, party) and prompt framing (direct elicitation, respondent embodiment, specialist embodiment). Across 811,560 prompts, the mean absolute error, representing the average difference between predicted and human response distributions, was 23 percentage points. The findings indicate that LLM responses diverge significantly when conditioned on demographics versus unconditioned prompts, and that model architecture choice impacts accuracy more than prompt framing. Refusal behavior also varied sharply by model and prompt design, underscoring LLMs' limitations for social simulation in sensitive domains.

Key takeaway

For research scientists or AI ethicists considering LLMs for social simulation, you should exercise extreme caution, especially with sensitive topics like suicide prevention. The observed 23 percentage point mean absolute error and significant demographic response divergence indicate current LLMs like GPT-5 Nano and Llama 3.1 8B are unreliable for reproducing nuanced public attitudes. Prioritize rigorous validation against human data and consider alternative methods before deploying LLMs in policy-critical or sensitive social modeling applications.

Key insights

LLMs struggle to accurately simulate social attitudes on sensitive topics, especially with demographic conditioning.

Principles

• LLM accuracy on sensitive topics is low.
• Model choice impacts accuracy more than prompt framing.
• Demographic conditioning alters LLM responses.

Method

The study systematically varied demographic conditioning, prompt framing, and model architecture across 811,560 prompts to evaluate LLM simulation of social attitudes.

In practice

• Evaluate LLM output for demographic bias.
• Test multiple LLM architectures for sensitive tasks.
• Be wary of LLM-generated public opinion data.

Topics

• Large Language Models
• Social Simulation
• Public Opinion Polling
• Suicide Prevention Policy
• Demographic Conditioning
• Model Evaluation Metrics

Best for: AI Scientist, Research Scientist, AI Ethicist

Related on AIssential

• I asked 5 different AIs to pick a number between 1 and 100… all of them said 42 😬 — LLMs exhibit cultural biases from training data, defaulting to non-random, culturally significant numbers when asked for randomness.
• Polarization by Default: Auditing Recommendation Bias in LLM-Based Content Curation — LLM-based content curation consistently amplifies polarization and exhibits demographic biases, often resistant to prompt engineering.
• Teaching Values to Machines: Simulating Human-Like Behavior in LLMs — LLMs can be induced to manifest human-like value structures, aligning strongly with human behavior patterns in simulations.
• Investigating Counterfactual Unfairness in LLMs towards Identities through Humor — LLMs exhibit counterfactual unfairness, applying stricter safety policies and attributing more malicious intent based on speaker-target identity.
• A Systematic Analysis of Biases in Large Language Models — LLMs, despite alignment efforts, inherit and perpetuate diverse biases from human-generated training data.
• Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBench — LLMs struggle to reconstruct specific, socially grounded consumer reactions in public discourse, despite strong technical performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.