When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals

2025-04-14 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

The study audits the reliability of self-consistency and cross-model agreement as confidence signals for Large Language Models (LLMs) in enterprise evaluation pipelines. Analyzing 265,000 samples from 53 runners across GPQA Diamond and AIME benchmarks, the research demonstrates that consistency is an unreliable indicator of correctness. Agreement is a positive but weak predictor, with Spearman ρ values ranging from 0.20 to 0.59. Its usefulness is regime-dependent, performing best for unsaturated mid-tier models and for compute allocation. Critically, the most consistent frontier model, gpt-4.1, exhibits over-confidence, with agreement ≥ 0.8 on 77% of GPQA cases, yet 48% of these are wrong, and it shows the lowest correlation with correctness (ρ=0.20). Chain-of-thought prompting improves accuracy but only marginally enhances the agreement-correctness signal. Confident errors frequently recur across different runners and prompts, and an exploratory cross-family check with Claude models suggests these errors are partly shared across providers, indicating shared bias rather than mere sampling noise.

Key takeaway

For MLOps Engineers designing LLM evaluation or routing pipelines, you should treat self-consistency as a coarse, cost-saving signal rather than a reliable confidence threshold. Avoid automatically trusting high-agreement answers from frontier models like gpt-4.1, as they often exhibit over-confidence and no accuracy advantage. Instead, prioritize mid-tier models for better agreement-correctness correlation and use consistency for adaptive sampling to reduce compute costs.

Key insights

LLM self-consistency is a weak, conditional proxy for correctness, not a standalone confidence signal, especially for frontier models.

Principles

• Agreement does not equal accuracy; shared bias can cause consistency.
• Self-consistency's utility varies by model tier and agreement saturation.
• Frontier models can be over-confident, degrading agreement as a signal.

Method

A large-scale cross-runner audit used K=50 samples per case, hierarchical runner-clustered bootstrap, and majority-correctness as the deployment label on GPQA Diamond and AIME.

In practice

• Use self-consistency for adaptive compute allocation (e.g., 60% sample reduction).
• Avoid routing hard cases to frontier models based on high agreement.
• Consider option-position bias in multiple-choice evaluations.

Topics

• LLM Evaluation
• Self-Consistency
• Model Confidence
• Frontier Models
• Adaptive Sampling
• LLM Bias

Code references

• dingkaihua/self_consistency_as_predictor_of_accuracy

Best for: Research Scientist, Machine Learning Engineer, NLP Engineer, AI Scientist, MLOps Engineer, AI Engineer

Related on AIssential

• When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals — Agreement among LLMs or within a model's samples is an unreliable, conditional proxy for correctness, not a standalone confidence signal.
• Accuracy, Stability, and Repeated-Run Reliability of Large Language Models on Deterministic Programming Tasks — LLM evaluation for deterministic tasks requires repeated-run stability analysis to complement conventional single-run accuracy reporting.
• Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models — LLMs widely violate statistical self-consistency, often performing better when aggregating fine-grained subpopulation estimates than direct population estimates.
• Self-Preference Is Weak or Absent in Verifiable Instruction-Following Revision: A Four-Model Test Under Genuine Authorship — LLMs show no detectable self-preference when revising their own verifiable instruction-following drafts, accepting machine-verified fixes at similar rates to fresh models.
• Multiagent Protocols with Aggregated Confidence Signals — Multiagent protocols aggregate individual LLM confidences into a single, discriminative system-level confidence, recovering performance losses from standard debate.
• Trust Me. I’m Artificial Intelligence. — AI models are becoming more convincing in their reasoning, but this does not equate to increased truthfulness or correctness.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.