Accuracy and Normalized Accuracy under Length Bias: Analysis, Guidelines, and a Bayesian Alternative

2026-07-14 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A new scoring rule, "Bayesian accuracy," addresses the pervasive length bias in multiple-choice benchmarks that rank candidate completions by conditional log-probability. Standard methods often penalize longer answers, while the common mitigation of normalizing scores by completion length frequently over-corrects, introducing a bias toward longer answers. Bayesian accuracy computes the posterior probability of each candidate under an explicit prior over answer length, effectively removing linear length effects. This method is a direct replacement for existing likelihood-based multiple-choice evaluation, requires no additional forward passes, and consistently demonstrates lower empirical length bias across various benchmarks and few-shot settings.

Key takeaway

For machine learning engineers evaluating language models on multiple-choice benchmarks, you should consider adopting Bayesian accuracy. This new scoring rule offers a more robust and less biased evaluation by explicitly accounting for answer length, unlike standard or length-normalized methods. Implementing it requires no extra forward passes, streamlining your evaluation workflow and providing more accurate insights into model performance without introducing artificial biases towards shorter or longer responses.

Key insights

Bayesian accuracy removes length bias in multiple-choice benchmarks via an explicit answer length prior.

Principles

• Standard accuracy penalizes longer answers.
• Length-normalized accuracy often over-corrects, biasing towards longer answers.
• Length biases depend on completion length distribution.

Method

Bayesian accuracy computes the posterior probability of each candidate under an explicit prior over answer length, removing linear length effects.

In practice

• Drop-in replacement for likelihood-based multiple-choice evaluation.
• Requires no additional forward passes.
• Consistently exhibits lower empirical length bias.

Topics

• Bayesian Accuracy
• Length Bias
• Multiple-Choice Benchmarks
• Language Model Evaluation
• Scoring Rules

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• CALIBER: Calibrating Confidence Before and After Reasoning in Language Models — Confidence in reasoning language models is state-dependent, requiring distinct supervision targets for pre- and post-reasoning estimates.
• BayesBench: Evaluating LLM Belief Trajectories Under Multi-Turn Evidence Accumulation — LLMs struggle to consistently translate improved latent inference into rational downstream predictions in multi-turn Bayesian tasks.
• Consistent but Miscalibrated: Evaluating LLM Limitations for Risk Communication in Natural Language — LLMs consistently misinterpret probabilistic information, particularly uncertainty, when generating natural language explanations, indicating a verbalization bottleneck.
• A Deep Dive into Calibration of Language Models: Platt Scaling, Isotonic Regression, Temperature Scaling — LLMs are often miscalibrated; post-hoc methods like Temperature Scaling, Platt Scaling, and Isotonic Regression can align confidence with accuracy.
• Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs — LLM uncertainty estimation in multilingual settings is significantly impacted by reasoning language and model scale.
• ConfidenceBench: Evaluating Confidence Calibration in Large Language Models — LLM confidence calibration is a distinct reliability metric, often diverging from accuracy, crucial for safe deployment.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.