Most LLM Conformity Needs No Speaker: Measuring the Speaker-Free Floor in Peer-Pressure Benchmarks

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

This research reveals that most apparent Large Language Model (LLM) "conformity"—where models change correct answers towards a peer response—persists even when the explicit speaker is removed. Standard conformity prompts conflate the presence of a speaker with the repetition of a wrong answer. Introducing a "no-source" condition, which removes the explicit speaker but retains the asserted wrong answer, the study found that six open-weight LLMs across seven datasets exhibited a 66.5% harmful revision rate in initially correct cases, significantly higher than the 10.3% from a plain re-ask. While expert-panel framing increased this rate by 12.9 percentage points to 79.4%, the speaker-free floor is the primary driver. The effect is robust to paraphrasing and open-ended settings, and models become confidently wrong after flipping. This highlights that repeated assertions, not just social influence, drive LLM revision.

Key takeaway

For AI Scientists and Machine Learning Engineers designing LLM conformity benchmarks or multi-agent systems, you must first establish a speaker-free baseline. Your evaluations may currently overstate social influence, as repeated assertions alone drive significant harmful revision (66.5%). Implement controls that remove explicit speakers to accurately measure source-attributed increments. Treat repeated or source-labeled assertions in untrusted contexts as potential manipulation surfaces, not independent evidence, to prevent models from confidently flipping to incorrect answers.

Key insights

LLM "conformity" is largely driven by repeated assertions, not explicit social influence, requiring a speaker-free baseline.

Principles

• LLM revision is primarily driven by repeated answer text.
• Source framing modulates, but does not create, revision.
• Agreement in multi-agent systems may not reflect independent evidence.

Method

A two-read arbitration protocol measures model answer shifts by inserting an asserted answer block, varying only its source framing, under greedy decoding.

In practice

• Measure speaker-free revision before attributing to social influence.
• Treat repeated assertions in untrusted contexts as manipulation surfaces.
• Compare inserted evidence against source-scrubbed paraphrases.

Topics

• LLM Conformity
• Speaker-Free Revision
• Prompt Design
• Multi-Agent Systems
• Model Evaluation
• Informational Influence

Code references

• yibo-hu-lab/llm-speaker-free-floor

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, Director of AI/ML

Related on AIssential

• Trust, but Don't Verify: Epistemic Blind Spots in LLM Source Evaluation — LLMs can detect fabricated statistics but fail to apply this capability during multi-source synthesis, prioritizing stylistic credibility.
• SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations — SoCRATES offers a multi-domain, socio-cognitively varied benchmark for reliable automated evaluation of proactive LLM mediators.
• The Cost of Consensus: Isolated Self-Correction Prevails Over Unguided Homogeneous Multi-Agent Debate — Unguided multi-agent LLM debate is costly and often degrades accuracy due to sycophancy and contextual fragility.
• Talking to a Know-It-All GPT or a Second-Guesser Claude? How Repair reveals unreliable Multi-Turn Behavior in LLMs — LLMs exhibit distinct and often unreliable conversational repair behaviors in multi-turn dialogues.
• A Large-Scale Multi-Dimensional Empirical Study of LLMs for Conversation Summarization — OmniCSEval offers a robust, multi-dimensional benchmark for evaluating LLMs in conversation summarization across diverse real-world scenarios.
• When large language models are reliable for judging empathic communication — LLMs reliably judge empathic communication, matching experts and outperforming crowdworkers when properly benchmarked.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.