LLM-as-a-Judge Scores Are Unreliable Optimization Signals in Closed-Loop Table Recognition

2026-07-20 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

The study investigates the reliability of LLM-as-a-judge signals for optimization in closed-loop table recognition, using FinTabNet (476 tables) and OmniDocBench (272 tables) datasets. It found that judge signals were weak, with scores frequently tying (37.5% to 49.1% of distinct-output pairs) and rankings being irreproducible. Only one selection policy, relying on an earliest-iteration tie rule, marginally beat random selection, but its advantage was not attributable to judge scores alone. The iterative loop often degraded quality on FinTabNet (TEDS fell from 0.792 to 0.772) and had a statistically neutral mean effect on OmniDocBench, concealing severe table-level losses (11.8% on FinTabNet, 4.4% on OmniDocBench). Degradation stemmed from "target-preservation failure" under unconstrained regeneration, where improvement instructions caused structural changes. A structure-preservation constraint significantly reduced severe losses on FinTabNet (from 3.6% to 0.8%) and showed directional consistency on OmniDocBench. The findings emphasize that LLM evaluation ability does not imply optimization utility.

Key takeaway

For MLOps Engineers deploying LLM-driven iterative refinement in table recognition, you should not rely solely on LLM-as-a-judge scores for optimization signals. Your safest baseline is often retaining the initial output (iter0). Implement prompt-level structure-preservation constraints and deterministic structural guards to mitigate severe degradation. Always conduct small, labeled audits to predict loop effects, as a neutral mean can hide significant table-level failures.

Key insights

LLM-as-a-judge evaluation ability does not reliably translate to optimization utility in closed-loop table recognition.

Principles

• Judge scores often tie, leading to irreproducible rankings.
• Unconstrained regeneration can cause structural degradation.
• Evaluation without GT differs from optimization with it.

Method

The study used an 8-iteration closed-loop image-to-HTML regeneration with Gemini 3.1 Flash Lite, comparing judge-feedback-based selection against random and oracle baselines, and testing structure-preservation constraints.

In practice

• Implement deterministic structural guards for HTML changes.
• Add prompt-level structure-preservation constraints.
• Conduct small labeled audits before deployment.

Topics

• LLM-as-a-Judge
• Table Recognition
• Iterative Refinement
• Closed-Loop Systems
• Evaluation Metrics
• Prompt Engineering
• Structural Preservation

Code references

• SWHL/TableRecognitionMetric

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• When the Judge Is Wrong — LLM-as-judge is unreliable for structured tasks, even with ground truth or source documents.
• Unlocking LLM Code Correction with Iterative Feedback Loops — Iterative feedback loops significantly enhance LLM code correction, especially for reasoning models and specific error types.
• When LLMs get significantly worse: A statistical approach to detect model degradations — A statistical framework using McNemar's test reliably detects LLM accuracy degradation by analyzing per-sample score differences.
• A Pelican Learns to Ride — Capturing every model iteration as a structured artifact makes benchmarks inspectable like code.
• When Gradients Collide: Failure Modes of Multi-Objective Prompt Optimization for LLM Judges — Multi-objective prompt optimization for LLM judges fails due to gradient dilution and instruction interference.
• Necessary but Not Sufficient: Temperature Control and Reproducibility in LLM-as-Judge Safety Evaluations — LLM-as-judge evaluations are not deterministic even at temperature 0, impacting safety assessment reproducibility.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.