What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

Research on Large Language Model (LLM) forecasters, specifically Eternis-Forecaster 8B, GLM-4.7-Flash, and GLM-4.5-Air on OpenForesight, reveals that while accurate, they often suffer from poor calibration and unfaithful Chain-of-Thought (CoT) reasoning. By training representation-pooling probes on intermediate activations, the study found substantially better calibration. These probes also act as "lie detectors," tracking behavioral shifts from prompt perturbations far more effectively than CoT, predicting the direction of change in 84% of cases. A key finding is that forecasts are largely fixed before reasoning begins; a single pre-reasoning pass can recover the committed answer and confidence. This insight enables routing questions by pre-set answer distribution, saving 30-47% of generated tokens without accuracy loss. The work establishes internal representation probing as a practical tool for calibrating, auditing, and triaging LLM forecasters and reasoning models.

Key takeaway

For AI Scientists deploying forecasting LLMs, relying solely on Chain-of-Thought for assessing model confidence or faithfulness is inadequate. You should integrate internal representation probing to achieve substantially better calibration and to accurately audit how evidence influences forecasts. This approach also allows you to identify pre-reasoning commitments, potentially saving 30-47% of generated tokens by optimizing question routing without sacrificing accuracy.

Key insights

Probing LLM internal representations reveals true forecast confidence and evidence sensitivity better than Chain-of-Thought.

Principles

• LLM forecasts are largely fixed before reasoning.
• CoT reasoning may not reflect underlying evidence.
• Internal representations improve forecast calibration.

Method

Train representation-pooling probes on intermediate activations to assess calibration and faithfulness, using evidence ablation and diversionary injection.

In practice

• Use probes for better forecast calibration.
• Audit LLM faithfulness via internal states.
• Route questions based on pre-reasoning confidence.

Topics

• Large Language Models
• Model Calibration
• Chain-of-Thought
• Internal Representations
• Forecasting
• Model Faithfulness

Best for: Research Scientist, AI Engineer, Machine Learning Engineer, AI Scientist

Related on AIssential

• What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness — Probing LLM internal representations offers superior calibration and faithfulness auditing compared to Chain-of-Thought.
• What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness — Probing LLM internal representations offers superior calibration and faithfulness insights compared to chain-of-thought.
• Reading and Steering Representations of Materials-Science Mechanisms in an Open-Weight Language Model — LLMs represent scientific mechanisms more reliably through state transformations than static states.
• Faithfulness Serum: Mitigating the Faithfulness Gap in Textual Explanations of LLM Decisions via Attribution Guidance — LLM-generated explanations often lack epistemic faithfulness, but a new method improves it via attention-level guidance.
• Building LMs that can model the world and model themselves | ARC Prize @ MIT — LLMs require explicit training for internal coherence, encompassing both factual consistency and accurate self-assessment of knowledge.
• Trained on Tokens, Calibrated on Concepts: The Emergence of Semantic Calibration in LLMs — Base LLMs exhibit emergent semantic calibration in Q&A, which instruction-tuning and chain-of-thought reasoning degrade.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.