🔮 Is AI immune to groupthink?

2020-12-30 · Source: Exponential View · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, medium

Summary

Rohit Krishnan's experiment investigates whether AI councils, composed of multiple Large Language Models (LLMs), exhibit groupthink similar to human committees. The study tested three configurations: a blending model, an LLM-council with peer review and a chairperson, and a "best answer" picker. Using 16 open-ended prompts, the findings reveal that LLM councils do not consistently preserve the best unique ideas from individual models. Specifically, the blended council retained only about 25% of useful, non-obvious ideas that originated from a single model. While peer review slightly favored shared ideas, keeping them about a third of the time compared to 24% for single-model ideas, it still significantly missed unique insights. This phenomenon mirrors the "biased sampling of shared information" observed in human group decision-making, indicating that while councils can improve average answers, they risk losing valuable, idiosyncratic contributions.

Key takeaway

For AI Engineers designing multi-agent LLM systems, recognize that simply combining model outputs can lead to groupthink and the loss of valuable, unique insights. You should implement explicit protocols to gather, store, and rank individual model ideas before final synthesis to prevent biased sampling of shared information. Experiment with different council structures for your specific problem sets, as a "one-size-fits-all" approach risks suboptimal outcomes and missed functionality.

Key insights

LLM councils, like human committees, can suffer from groupthink, losing unique valuable ideas despite improving average answers.

Principles

• Model diversity improves LLM responses.
• Group decision-making can filter unique insights.
• Explicit protocols are crucial for LLM council design.

Method

The experiment involved setting up LLM committees (blending, peer review, direct pick), breaking answers into "cards" using Sonnet, clustering similar cards, and having two judges score solo-derived clusters blindly.

In practice

• Design LLM councils to explicitly capture unique ideas.
• Experiment with council structures for specific problems.
• Store and rank individual model ideas before final synthesis.

Topics

• LLM Councils
• Groupthink
• Multi-agent Systems
• Model Diversity
• AI Experimentation
• Information Bias

Code references

• karpathy/llm-council
• strangeloopcanon/ai_committee_brain

Best for: Research Scientist, AI Architect, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity — LLMs are more easily misled by peer consensus than corrected, even with reasoning interventions.
• Complacent, Not Sycophantic: Reframing Large Language Models and Designing AI Literacy for Complacent Machines — LLM behavior is better understood as "complacency" driven by design, not "sycophancy" implying intent.
• Diversity Collapse in Multi-Agent LLM Systems: Structural Coupling and Collective Failure in Open-Ended Idea Generation — Multi-agent LLM systems for ideation suffer diversity collapse due to interaction structures, not model insufficiency.
• Everyone Building Multi-Agent Systems Is Spending Compute on Something Mathematically Impossible. — Multi-agent systems inherently lose information at each inter-agent handoff, making them mathematically inferior to single agents under equal compute.
• Week Ending 3.22.2026 — Advancements in LLM reasoning, multi-agent coordination, creativity, and social choice highlight diverse progress in AI capabilities and understanding.
• Network Effects and Agreement Drift in LLM Debates — LLM agent opinion dynamics are shaped by network structure, group imbalance, and a systematic "agreement drift" bias.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Exponential View.