Calibrating Scientific Foundation Models with Inference-Time Stochastic Attention

2026-04-22 · Source: stat.ML updates on arXiv.org · Field: Science & Research — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences, Environmental Science & Earth Systems · Depth: Expert, quick

Summary

A new method called Stochastic Attention has been proposed to improve the calibration and predictive uncertainty of Transformer-based scientific foundation models. This lightweight, inference-time modification replaces softmax weights with normalized multinomial samples, controlled by a single concentration parameter, to generate predictive ensembles without requiring model retraining. The authors introduce a calibration objective to efficiently tune this parameter post-hoc, taking only minutes compared to days for competitive baselines. Evaluated on scientific foundation models for weather and timeseries forecasting, plus an additional regression task, Stochastic Attention demonstrated superior native calibration and sharper prediction intervals at comparable coverage against existing uncertainty-aware methods.

Key takeaway

For AI Scientists and Machine Learning Engineers deploying scientific foundation models in high-stakes environments, integrating Stochastic Attention can significantly enhance predictive uncertainty calibration and interval sharpness. This method offers a time-efficient alternative to extensive retraining, requiring only minutes of post-hoc tuning to achieve robust uncertainty quantification, thereby improving model trustworthiness and reliability in critical applications.

Key insights

Stochastic Attention improves scientific foundation model uncertainty calibration via a lightweight, inference-time modification.

Principles

• Randomize attention for uncertainty.
• Calibrate post-hoc, not via retraining.
• Match stochastic output to target.

Method

Stochastic Attention randomizes Transformer attention weights using normalized multinomial samples controlled by a concentration parameter. This parameter is tuned post-hoc via a calibration objective that matches the stochastic output to the target, generating predictive ensembles.

In practice

• Apply to Transformer-based models.
• Use for weather forecasting.
• Enhance timeseries predictions.

Topics

• Scientific Foundation Models
• Stochastic Attention
• Predictive Uncertainty
• Inference-Time Calibration
• Transformer Architectures

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• How Confident Is the First Token? An Uncertainty-Calibrated Prompt Optimization Framework for Large Language Model Classification and Understanding — Calibrating LLM first-token uncertainty with label priors enables efficient, adaptive prompt optimization for classification.
• Attention Calibration for Position-Fair Dense Information Retrieval — Inference-time attention calibration, especially partial application, effectively reduces positional bias in dense retrieval models without retraining.
• [AINews] Cognition raises $1B in $26B Series D — AI development is rapidly maturing, driven by architectural inference optimizations, agentic system integration, and specialized model releases.
• Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation — Principled initialization using teacher statistics dramatically improves hybrid linear attention model distillation efficiency.
• Forecasting Scientific Progress with Artificial Intelligence — Current AI systems cannot reliably forecast scientific progress or its timing, despite identifying plausible research directions.
• Adaptive Conformal Prediction for Improving Factuality of Generations by Large Language Models — Adaptive conformal prediction improves LLM factuality by calibrating uncertainty thresholds based on prompt characteristics, enhancing conditional coverage.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.