When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals

2026-07-09 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A large-scale cross-runner study involving 53 participants and 265,000 samples across GPQA Diamond and AIME datasets challenges the assumption that consistency in LLM responses indicates correctness. The research, which drew K=50 samples per case, found that both self-consistency and cross-model agreement are unreliable indicators of accuracy. While agreement is a positive but weak predictor (rho 0.20-0.59), its utility varies significantly. It proves most effective for unsaturated mid-tier models and for allocating computational resources. Conversely, the study reveals that frontier models exhibit over-confidence, with agreement scores >=0.8 on 77% of GPQA case-result entries, yet 48% of these highly consistent responses were incorrect. An exploratory check on three Claude tiers corroborated this pattern of confident errors recurring across providers. Therefore, self-consistency serves as a conditional proxy for correctness, not a standalone confidence signal.

Key takeaway

For machine learning engineers evaluating AI systems using LLM-as-judge, you should critically assess consistency signals. Do not solely rely on self-consistency or cross-model agreement as a confidence score, especially with frontier models, which exhibit over-confident errors. Instead, consider agreement as a conditional proxy, most useful for mid-tier models or for optimizing compute allocation. Your evaluation pipelines require more robust correctness indicators beyond mere agreement.

Key insights

Agreement among LLMs or within a model's samples is an unreliable, conditional proxy for correctness, not a standalone confidence signal.

Principles

• LLM consistency does not guarantee accuracy.
• Shared bias can drive agreement, not truth.
• Agreement utility is regime-dependent.

Method

A large-scale cross-runner study drew K=50 samples for overlapping cases across model tiers, prompting, and scale on GPQA Diamond and AIME, using majority-correctness as the deployment label and hierarchical runner-clustered bootstrap.

In practice

• Prioritize agreement for mid-tier models.
• Use agreement for compute allocation.
• Distrust high agreement in frontier models.

Topics

• LLM Evaluation
• Self-Consistency
• Cross-Model Agreement
• Model Confidence
• Frontier Models
• GPQA Diamond
• AIME Dataset

Best for: Research Scientist, AI Architect, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals — LLM self-consistency is a weak, conditional proxy for correctness, not a standalone confidence signal, especially for frontier models.
• Evaluating AI Agents: Can LLM‑as‑a‑Judge Evaluators Be Trusted? — LLM-as-a-Judge offers scalable, private AI agent evaluation, but its reliability hinges on human alignment, self-consistency, and inter-model agreement.
• Trust Me. I’m Artificial Intelligence. — AI models are becoming more convincing in their reasoning, but this does not equate to increased truthfulness or correctness.
• Validity of LLMs as data annotators: AMALIA on authority — LLM agreement with human annotators does not guarantee construct validity; inferential path matters.
• Trust, but Don't Verify: Epistemic Blind Spots in LLM Source Evaluation — LLMs can detect fabricated statistics but fail to apply this capability during multi-source synthesis, prioritizing stylistic credibility.
• Can We Trust LLM's Logic? Quantifying Uncertainty, Coherence, and Robustness via a Graph-Based Framework — GRAPHEVAL quantifies LLM reasoning uncertainty and improves fidelity by analyzing intermediate steps via graph-based coherence and a medoid-based decoding strategy.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.