Point of Order: Action-Aware LLM Persona Modeling for Data-Grounded Civic Deliberation

2026-07-13 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Natural Language Processing · Depth: Expert, extended

Summary

A new reproducible pipeline converts public Zoom recordings into speaker-attributed transcripts, addressing the limitation of anonymous speaker labels in ASR data for multi-party deliberation simulations. This method integrates visual cues, audio features, and textual context to assign stable identities across videos, then augments transcripts with metadata like persona profiles and pragmatic action tags (e.g., [propose_motion]). Three new datasets of local government deliberations—Appellate Court hearings, School Board meetings, and Municipal Council sessions, totaling ~1.72M words—are released. Fine-tuning LLMs with this "action-aware" data significantly improves performance, achieving a 67% reduction in perplexity and nearly doubling classifier-based metrics for speaker fidelity and realism. Human evaluations show these simulations are often indistinguishable from real deliberations, with temporal anchoring further enhancing topic coverage and goal achievement.

Key takeaway

For Machine Learning Engineers developing multi-agent dialogue systems or civic simulations, this work offers a validated framework to achieve unprecedented realism. You should integrate multimodal speaker-linking and action-aware LLM fine-tuning with structured metadata to create highly authentic persona models. This approach significantly improves simulation fidelity and coherence, but be aware of the ethical implications regarding potential misuse for misinformation or manipulation in high-stakes public discourse.

Key insights

Speaker-attributed, action-aware data significantly enhances LLM persona modeling for realistic multi-party civic simulations.

Principles

• Structured metadata, particularly action tags, is crucial for simulation fidelity.
• Prompt complexity should align with model training: rich for baseline, concise for fine-tuned.
• Temporal anchoring improves agenda coherence and deliberative goal achievement.

Method

A pipeline extracts speaker identities from Zoom videos via OCR, transcribes audio with Whisper, then uses GPT-5 to generate persona profiles, topics, and action tags for PEFT-based LLM fine-tuning.

In practice

• Integrate visual and audio cues for robust speaker identification in multi-party videos.
• Prepend action tags to utterances to guide LLMs toward pragmatic functions.
• Implement time-aware prompting to improve topic coverage and decision-making in simulations.

Topics

• LLM Persona Modeling
• Multi-Agent Simulation
• Speaker Diarization
• Action Tagging
• Civic Deliberation
• Parameter-Efficient Fine-Tuning

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• LLM-enabled Social Agents — Persona-based role definitions are crucial for LLM-enabled social agents to achieve socially intelligible and coherent behavior.
• LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokens — LLaMo unifies motion-language understanding and generation in LLMs using continuous tokens and a MoT architecture.
• Adaptive Interviewing for Persona Simulation in LLMs: Evidence-Grounded Reasoning Improves Decision Alignment — Adaptive interviewing improves LLM decision simulation by selectively grounding predictions in user-specific, follow-up derived evidence.
• The Story Shapes the Agent: Narrative Priors in LLM Behavior — Narrative framing significantly shapes LLM agent behavior, often overriding explicit persona instructions.
• Digital Pantheon: Simulating and Auditing Coalition Formation with LLM Agents — LLM agents can simulate partisan political coalition formation with auditable lineage and real-world grounding.
• From Reactive to Proactive: Assessing the Proactivity of Voice Agents via ProVoice-Bench — ProVoice-Bench evaluates proactive voice agents, revealing significant performance gaps in current Multimodal LLMs.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.