The Silent Vote: Improving Zero-Shot LLM Reliability by Aggregating Semantic Neighborhoods

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A new inference-time layer, Semantic Softmax, significantly improves zero-shot Large Language Model reliability by addressing Renormalization Bias and the "Silent Vote" phenomenon. This bias occurs when standard constrained decoding in LLMs discards probability mass from semantic synonyms, leading to artificial overconfidence and poor calibration. Semantic Softmax recovers this lost information by aggregating scores from the semantic neighborhood of each target label. Evaluated on Qwen-3 and Phi-4-mini models using the GoEmotions and Civil Comments datasets, the approach consistently reduces Expected Calibration Error (ECE) and Brier Score. Simultaneously, it enhances discriminative performance, measured by AUROC and Macro-F1, offering a more calibrated and accurate alternative for zero-shot classification by accounting for linguistic nuances.

Key takeaway

For Machine Learning Engineers deploying zero-shot LLM classifiers, consider integrating Semantic Softmax to significantly enhance model reliability and calibration. Your models, especially Qwen-3 and Phi-4-mini, will exhibit reduced Expected Calibration Error and Brier Score, alongside improved AUROC and Macro-F1. This method directly addresses Renormalization Bias, ensuring your zero-shot classifications are more accurate and trustworthy by accounting for linguistic nuances.

Key insights

Semantic Softmax improves zero-shot LLM classification by aggregating semantic synonym scores, mitigating Renormalization Bias and enhancing calibration.

Principles

• Renormalization Bias causes artificial overconfidence in LLMs.
• Semantic synonyms hold "Silent Vote" information.
• Aggregating semantic neighborhoods improves calibration.

Method

Semantic Softmax is an inference-time layer that recovers lost probability mass by aggregating scores of semantic neighborhoods surrounding each target label in constrained decoding.

In practice

• Apply Semantic Softmax to zero-shot classifiers.
• Use Qwen-3 or Phi-4-mini for evaluation.
• Improve calibration and discriminative performance.

Topics

• Zero-shot Classification
• Large Language Models
• Semantic Softmax
• Renormalization Bias
• Model Calibration
• Qwen-3
• Phi-4-mini

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Zero-Shot Goal Recognition with Large Language Models — LLM goal recognition competence varies significantly, with some models integrating evidence effectively while others rely on world-knowledge priors.
• Addressing Data Scarcity in Bangla Fake News Detection: An LLM-Based Dataset Augmentation Approach — LLM-based data augmentation significantly improves fake news detection in low-resource languages by generating diverse synthetic data.
• Isolating LLM Lexical Bias: A Curation-Free Triangulated Metric for Preference-Stage Learning — A new metric quantifies LLM lexical bias from preference learning without manual curation.
• Clustered Self-Assessment: A Simple yet Effective Method for Uncertainty Quantification in Large Language Models — The Clustered Self-Assessment method quantifies LLM uncertainty by structuring semantically clustered generations into multiple-choice questions for self-evaluation.
• Formalizing and Mitigating Structural Distortion in LLM Attention for Zero-Shot Graph Reasoning — LLM performance on graphs is bottlenecked by structural distortion from sequence linearization, correctable by attention alignment.
• Adaptive Conformal Prediction for Improving Factuality of Generations by Large Language Models — Adaptive conformal prediction improves LLM factuality by calibrating uncertainty thresholds based on prompt characteristics, enhancing conditional coverage.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.