The Necessity of Setting Temperature in LLM-as-a-Judge

2026-06-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

A systematic investigation reveals the critical impact of temperature settings on LLM-as-a-Judge performance, challenging the empirical convention of fixed temperature choices. Researchers from the University of Luxembourg and ETH Zürich conducted over 180,000 inference calls per model across six temperature settings ([0.01, 0.5, 1.0, 1.5, 2.0, 3.0]) using models like Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 and google/gemma-3-27b-it. Findings show low temperatures (T=0.01) result in near-perfect consistency (≈ 1.00) and negligible error rates (≈ 0.00). Conversely, high temperatures (T=3.0) drastically reduce consistency (e.g., Qwen3-Next-80B to 0.57) and significantly increase error rates (up to 0.49). Causal analysis confirms temperature structurally influences judge behavior, primarily undermining stability, reproducibility, and increasing format parsing errors, rather than directly impacting accuracy. High temperatures can foster deeper reasoning but also lead to instruction non-compliance and indecisiveness.

Key takeaway

For Machine Learning Engineers configuring LLM-as-a-Judge systems, your temperature setting is not a trivial detail but a core determinant of evaluation reliability. You should move beyond fixed low-temperature defaults and adopt task-adaptive strategies. For high-stakes, deterministic evaluations, prioritize low temperatures to ensure consistency and minimize errors. However, if your goal is deeper, more human-like reasoning, use higher temperatures, potentially combining a high-temperature agent for initial judgment with a low-temperature agent for output correction.

Key insights

LLM-as-a-Judge performance is critically sensitive to temperature, affecting consistency, error rates, and reasoning depth.

Principles

• Low temperatures ensure high consistency and minimal errors.
• High temperatures foster deeper reasoning but increase errors.
• Temperature effects vary by model, prompt, and judge type.

Method

A causal inference framework, using Cross-Fitted AIPW with LightGBM regressors, rigorously examines temperature's direct effect on LLM judge behavior.

In practice

• Prioritize low temperatures for deterministic evaluations.
• Use high temperatures for exploratory, nuanced judgments.
• Combine high-T agents for reasoning, low-T for correction.

Topics

• LLM-as-a-Judge
• Decoding Temperature
• Causal Inference
• Model Evaluation
• Prompt Engineering
• LLM Consistency

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most — More capable LLMs make worse forecasts for superlinear growth and tail risk, especially in the upper tail.
• GPT-5 Killed Temperature Control. Most People Haven’t Noticed… — GPT-5 removes statistical sampling controls, shifting to semantic guidance for reliable, multi-stage inference.
• Judge the Judge: Building LLM Evaluators That Actually Work with GEPA — Mahmoud Mabrouk, Agenta AI — Calibrated LLM judges, aligned with human annotations via prompt optimization, accelerate AI agent development and reliable online evaluation.
• The Statistics of Token Selection: Logits, Temperature, and Top-P Walkthrough — LLM output generation relies on a statistical pipeline involving logits, temperature, and top-p for next-token prediction.
• Month in 4 Papers (April 2026) — LLM accuracy can unexpectedly improve with "rude" prompts compared to "polite" ones.
• How to Reduce LLM Inference Cost and Improve Accuracy with Pass@k and Majority Voting — Random sampling at inference can be leveraged for accuracy gains via multiple attempts, especially with cost-optimized models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.