A Model of Multi-turn Human Persuadability Using Probabilistic Belief Tracing

2025-08-07 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Social Sciences & Behavioral Studies · Depth: Expert, extended

Summary

PersuasionTrace is a novel framework and web-based experimental platform designed to analyze multi-turn human-LLM persuasion dynamics, moving beyond traditional pre/post belief change measures. It records human belief trajectories across dialogue turns, annotates LLM persuader messages with rhetorical dimensions (logos, pathos, ethos), and evaluates simulated targets based on their fidelity to human belief updates. Using this framework, researchers found that human targets exhibit two distinct belief update patterns and differential susceptibility to rhetorical strategies, with ethos negatively correlating with persuasion delta. The study also revealed that LLMs, such as gpt-5-2025-08-07, are persuasive across diverse topics and modalities. Critically, the paper introduces a Bayesian-network simulated target that achieves human-like belief dynamics (81 vs 80 for human reference), significantly outperforming baseline LLM simulators (64). This process-level evaluation highlights how simulator choice impacts persuader policy rankings, advocating for fidelity-based assessment for safer AI system optimization.

Key takeaway

For AI Scientists and Research Scientists developing or evaluating persuasive AI, relying solely on pre/post belief change metrics is insufficient and risky. You should integrate process-level evaluation, such as multi-turn belief tracing and rhetorical analysis, to understand how persuasion unfolds. This approach, coupled with human-grounded simulators like the Bayesian-network model, will prevent optimizing systems against non-human-like belief dynamics, ensuring safer and more robust persuasive AI.

Key insights

Process-level belief tracing and human-faithful simulators are crucial for understanding and safely optimizing LLM persuasion.

Principles

• Endpoint measures alone obscure persuasion mechanisms.
• Human belief dynamics are heterogeneous and rhetorically sensitive.
• Simulator fidelity to human trajectories is paramount.

Method

PersuasionTrace uses a web platform for multi-turn belief reports, LLM-based rhetorical annotation, and a Bayesian-network simulator with atomization, state update, and verbalization.

In practice

• Implement multi-turn belief elicitation in persuasion studies.
• Annotate rhetorical appeals (logos, pathos, ethos) in dialogues.
• Develop structured simulators with explicit latent belief states.

Topics

• LLM Persuasion
• Belief Tracing
• Bayesian Networks
• Rhetorical Strategies
• AI Safety
• Human-AI Interaction

Code references

• jlcmoore/persuasiontrace

Best for: AI Scientist, Research Scientist, AI Ethicist

Related on AIssential

• A Model of Multi-turn Human Persuadability Using Probabilistic Belief Tracing — PERSUASIONTRACE offers a framework and Bayesian target for analyzing multi-turn human-LLM persuasion dynamics with process fidelity.
• ‘Probably’ doesn’t mean the same thing to your AI as it does to you — AI chatbots misinterpret words of estimative probability compared to humans, posing risks in high-stakes applications.
• Imperfectly Cooperative Human-AI Interactions: Comparing the Impacts of Human and AI Attributes in Simulated and User Studies — AI transparency significantly impacts human trust and satisfaction in imperfectly cooperative interactions, often more than human personality.
• AI chatbots can effectively sway voters – in either direction — AI chatbots effectively sway political opinions by generating numerous factual claims, even if some are inaccurate.
• Understanding Persuasion in Long-Running Agents — Persistent belief conditioning, not transient persuasion, significantly alters LLM agent behavior in downstream tasks.
• Language Models Exhibit Inconsistent Biases Towards Algorithmic Agents and Human Experts — LLMs exhibit inconsistent biases, preferring human experts in stated trust but algorithms in revealed preference tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.