Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

Small language models (LMs) can serve as efficient, reference-free evaluators by utilizing their internal representations, a new paradigm called "Representation-as-a-Judge." This approach, instantiated by the INSPECTOR framework, addresses the high cost, opacity, and prompt sensitivity of the "LLM-as-a-Judge" method. Researchers found that small LMs, despite weak generative abilities, encode rich evaluative signals in their hidden states, supporting the Semantic Capacity Asymmetry Hypothesis. INSPECTOR extracts aspect-level evaluation scores from a strong LLM judge (DeepSeek-V3), then trains lightweight classifiers on internal representations from small LMs like Qwen3-1.7B or Llama-3.2-1B-Instruct. Experiments on reasoning benchmarks (GSM8K, MATH, GPQA) show INSPECTOR significantly outperforms prompting-based small LMs, achieving over 20% higher F1 scores, and closely approximates full LLM judges. Its binary classification performance, reaching 80-90% F1, makes it a reliable data filter for supervised fine-tuning.

Key takeaway

For Machine Learning Engineers curating datasets or evaluating model outputs, you should consider adopting the "Representation-as-a-Judge" paradigm. This method allows you to use smaller, more efficient LMs like Qwen3-1.7B to approximate the evaluation quality of large LLM judges, reducing computational costs and improving interpretability. Implement probing-based classifiers, particularly with mean pooling and Logistic Regression, to filter high-quality data for supervised fine-tuning, as this approach significantly outperforms direct prompting and random selection.

Key insights

Small LMs encode strong evaluative signals in internal representations, enabling efficient "Representation-as-a-Judge" evaluation.

Principles

• Evaluation requires less semantic capacity than generation.
• Evaluative signals reside in intermediate representations.
• Scaling laws don't always apply to evaluation performance.

Method

INSPECTOR obtains LLM-judge scores, extracts small LM internal representations (mean, last, min, max, concat pooling, attention/statistical features), and trains lightweight classifiers (e.g., Logistic Regression) on these features.

In practice

• Use probing for efficient, reference-free evaluation.
• Filter noisy reasoning data for SFT.
• Employ mean pooling and Logistic Regression for probes.

Topics

• LLM Evaluation
• Small Language Models
• Representation Probing
• Semantic Capacity Asymmetry
• Data Filtering
• Supervised Fine-Tuning

Code references

• zhuochunli/Representation-as-a-judge
• hendrycks/math

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data — Base LLMs inherently predict external judge scores, a latent ability efficiently elicited by Self-Evaluation Elicitation (SEE) with minimal data.
• Large Language Models Do Not Always Need Readable Language — LLMs can recover semantics from compact, non-human-readable textual representations, decoupling readability from model-side understanding.
• SLMJury: Can Small Language Models Judge as Well as Large Ones? — Small Language Models can effectively judge model outputs, but their performance is highly dependent on the task domain and judging paradigm.
• Challenges and Recommendations for LLMs-as-a-Judge in Multilingual Settings and Low-Resource Languages — LLM-as-a-Judge faces significant reliability issues and overtrust in multilingual and low-resource language evaluation due to proficiency gaps.
• I Built a RAG System. Then I Added Evals. Here’s What I Found. — Rigorous LLM-as-a-judge evaluation is crucial for RAG system reliability, exposing hidden data and prompt issues.
• The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs — Quantization in LLMs creates behavioral divergence not captured by accuracy, necessitating new evaluation metrics.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.