On the sensitivity of machine-learned probabilistic weather forecast models to scale-aware scoring rules

2026-07-22 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Environmental Science & Earth Systems · Depth: Expert, extended

Summary

A preliminary study compared machine-learned probabilistic weather forecast models, specifically versions of AIFS-CRPS, trained with various univariate and multivariate scoring rules designed for scale-awareness. The research, conducted using an O96 (approximately 1-degree) model within the Anemoi framework and ERA5 reanalysis data from 1979-2020, found broadly similar forecast skill across standard verification metrics. The graph energy score setup performed somewhat better in the tropics, while the global energy score showed some degradation compared to the almost fair Continuous Ranked Probability Score (CRPS). Crucially, explicit scale-awareness, whether through multi-scale loss formalisms or spectral scores, significantly improved the realism of forecast field spectra, particularly for small-scale variability. The study highlighted that the weighting of different fields, scales, and variables is a more critical factor for spectral fidelity than the specific multivariate scoring mechanism chosen.

Key takeaway

For AI Scientists developing probabilistic weather forecast models, consider integrating localized multivariate scoring rules like the graph energy score into your training objectives. You should prioritize explicit scale-awareness in your loss functions, as this significantly improves the physical realism of forecast field spectra. Furthermore, carefully tune the weighting of different fields, scales, and variables within your scale-aware losses, as this factor often outweighs the choice of the specific multivariate score for achieving spectral fidelity.

Key insights

Localized multivariate scoring rules and explicit scale-awareness improve machine-learned probabilistic weather forecasts, especially spectral realism.

Principles

• Multivariate scores are viable CRPS alternatives.
• Scale-aware losses enhance forecast field realism.
• Weighting scales impacts spectral fidelity significantly.

Method

Train machine-learned probabilistic weather models using loss functions derived from univariate (CRPS) or multivariate (energy, graph energy) scoring rules. Introduce scale-awareness via multi-scale loss formalisms or spectral transformations.

In practice

• Use graph energy scores for tropical weather forecasts.
• Apply scale-aware loss to improve small-scale variability.
• Implement variable-dependent weighting for spectral losses.

Topics

• Probabilistic Weather Forecasting
• Machine Learning Models
• Scoring Rules
• Scale-Aware Loss
• Graph Neural Networks
• Spectral Analysis

Code references

• ecmwf/anemoi-core

Best for: AI Scientist, Research Scientist

Related on AIssential

• Improving the sharpness in neural network-based parametric post-processing of ensemble forecasts — A penalty term in neural network loss functions can improve forecast sharpness without sacrificing accuracy.
• The weather and climate science AI revolution isn’t revolutionary — Machine learning enhances weather and climate modeling through efficiency and targeted applications, not by replacing core physics.
• [D] Conformal Prediction vs naive thresholding to represent uncertainty — Conformal Prediction offers theoretical guarantees for uncertainty quantification, unlike naive thresholding.
• A Practitioner's Guide to Kolmogorov-Arnold Networks — KANs offer enhanced expressivity and interpretability over MLPs by using learnable univariate basis functions on network edges.
• Toward Simultaneously Optimal Regret in U-Calibration — A new U-calibration algorithm achieves simultaneously optimal regret rates for both general and smooth proper loss functions.
• Performance Evaluation of GraphCast for Medium-Range Weather Forecasting over Brazil — GraphCast's skill in tropical regions is regime-dependent, excelling in extended range and large-scale moisture.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.