Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biases

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, short

Summary

A systematic comparison by Huang, Wu, Yang, and Arase (2026) investigates Large Reasoning Models (LRMs) as superior judges compared to non-reasoning Large Language Models (LLMs). Published in the Proceedings of the Workshop on Evaluating Evaluations (EvalEval) in July 2026, their empirical analysis reveals four key findings. LRMs demonstrate higher judgment accuracy, particularly for reasoning-intensive tasks, and exhibit superior instruction-following capabilities. They also show enhanced robustness against adversarial attacks targeting judgment tasks. However, a significant drawback is that LRMs still suffer from strong evaluation biases. To address this, the authors propose PlanJudge, a lightweight evaluation strategy that prompts the model to generate an explicit evaluation plan before making a judgment. Experiments indicate PlanJudge effectively mitigates these biases in LLM-as-a-Judge scenarios while maintaining overall judgment accuracy.

Key takeaway

For Machine Learning Engineers evaluating LLM outputs, you should consider Large Reasoning Models (LRMs) as superior judges for accuracy and instruction-following. This applies especially to complex reasoning tasks. However, be aware of their inherent evaluation biases. Implement the PlanJudge strategy by prompting your LRM to create an explicit evaluation plan first. This simple step can significantly reduce bias in your LLM-as-a-Judge setup while maintaining high judgment accuracy.

Key insights

Large Reasoning Models are better LLM-judges but require bias mitigation through explicit planning.

Principles

• LRMs excel in reasoning-intensive judgment tasks.
• LRMs are more robust to adversarial attacks.
• Evaluation biases persist in LRMs.

Method

PlanJudge prompts the model to generate an explicit evaluation plan before executing judgment, significantly mitigating biases while preserving accuracy.

In practice

• Implement PlanJudge for LLM-as-a-Judge evaluations.
• Prioritize LRMs for complex reasoning tasks.

Topics

• Large Reasoning Models
• LLM-as-a-Judge
• Evaluation Bias
• PlanJudge
• Adversarial Robustness
• Instruction Following

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models — LRMs exhibit a significant "production-evaluation gap" due to answer confirmation bias, struggling to evaluate flawed reasoning despite correct answers.
• SLMJury: Can Small Language Models Judge as Well as Large Ones? — Small Language Models can effectively judge model outputs, but their performance is highly dependent on the task domain and judging paradigm.
• When to Think Deeply: Inhibitory Deliberation for LLM Reasoning — Selective, response-conditioned inhibitory deliberation significantly enhances LLM reasoning efficiency and accuracy.
• Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training — Reasoning LLM-as-judges outperform non-reasoning judges in non-verifiable LLM post-training, despite generating adversarial outputs.
• Benchmarks in Leipzig — Iterative, multi-stage evaluation reveals impressive mathematical reasoning capabilities in Large Language Models.
• The Necessity of Setting Temperature in LLM-as-a-Judge — LLM-as-a-Judge performance is critically sensitive to temperature, affecting consistency, error rates, and reasoning depth.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.