LLM Judges Can Be Too Generous When There Is No Reference Answer

2026-07-07 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, extended

Summary

The study "LLM Judges Can Be Too Generous When There Is No Reference Answer" investigates the reliability of LLM judges in evaluating open-ended model responses, particularly in reference-free settings. Researchers developed a two-stage pipeline involving calibration and sensitivity experiments across three languages (English, Arabic, Telugu) and two datasets (TyDi QA, MATA). Findings indicate that LLM judges tend to over-credit incorrect answers without a reference, with adding reference information flipping decisions by up to 85% in some cases. This effect is more pronounced in lower-resource languages like Telugu. Human annotation comparisons confirm that reference-driven changes generally align with human judgments, highlighting the need for calibration before deploying LLM judges in reference-free scenarios.

Key takeaway

For AI Scientists and Machine Learning Engineers deploying LLM judges for open-ended response evaluation, you must perform a two-stage calibration and sensitivity analysis. This process helps identify suitable LLM judges by revealing their tendency to over-credit in reference-free settings, especially for lower-resource languages. Incorporate a sample of ground-truth answers into your evaluation pipeline. This ensures judge reliability and aligns verdicts more closely with human judgments before scaling to reference-free scenarios.

Key insights

LLM judges over-credit responses in reference-free settings, requiring calibration with ground-truth data for reliability.

Principles

• LLM judges over-credit without references.
• Reference visibility significantly alters verdicts.
• Calibration improves judge reliability.

Method

A two-stage pipeline: (a) calibration experiments assess judge knowledge of the task, and (b) sensitivity experiments measure impact of reference presence/position on judge performance.

In practice

• Calibrate LLM judges with gold-standard answers.
• Conduct sensitivity analysis for reference-free tasks.
• Use a JSON output format for judge verdicts.

Topics

• LLM Evaluation
• LLM-as-a-Judge
• Reference-Free Evaluation
• Multilingual NLP
• Calibration Techniques
• Bias Detection

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Your LLM Judge Is Not Measuring What You Think It Is — Uncalibrated LLM judges introduce systemic biases like verbosity and position preference, requiring explicit calibration for reliable evaluation.
• Reference-Free Evaluation of Reasoning in Open-Ended Question Answering — A hypergraph-based NLI audit provides more reliable, reference-free LLM reasoning evaluation than direct LLM judges, especially in high-stakes domains.
• Challenges and Recommendations for LLMs-as-a-Judge in Multilingual Settings and Low-Resource Languages — LLM-as-a-Judge faces significant reliability issues and overtrust in multilingual and low-resource language evaluation due to proficiency gaps.
• Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data — Base LLMs inherently predict external judge scores, a latent ability efficiently elicited by Self-Evaluation Elicitation (SEE) with minimal data.
• References Improve LLM Alignment in Non-Verifiable Domains — Reference-guided LLM evaluators significantly improve model alignment and self-improvement in non-verifiable domains.
• LLM Judges Can Be Too Generous When There Is No Reference Answer — LLM judges over-credit incorrect answers without reference answers, requiring calibration for reliable evaluation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.