Building a Local LLM-as-Judge Pipeline for Image Dataset Curation

2026-07-08 · Source: HackerNoon · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Intermediate, long

Summary

A local LLM-as-judge pipeline, named MonkeyEatingMango, was developed to curate image datasets for a travel app, specifically addressing errors like "celebrity matches" in automatically sourced photos. This system uses a Lambda Architecture, separating a fast, approximate "hot path" from a slower, corrective "cold path" that runs locally on an NVIDIA GeForce RTX 3060 with 12GB vRAM. The cold path employs `gemma3:12b` for two tasks: classifying place names as photographable venues and judging if candidate images correctly represent them. A `qwen2.5vl, 3B` model handles initial coarse visual checks, running three times faster for simple "place or not" decisions. The pipeline re-evaluated ~17,000 unchecked photos, recovering correct images for about 30% of initially photoless entries, improving accuracy from 36% to 65% for findable photos. It also features a "negative cache" and an "andon cord" for manual review. String hygiene issues, not photo scarcity, were the primary cause of initial lookup failures.

Key takeaway

For MLOps Engineers building robust data curation pipelines for visual assets, consider adopting a Lambda Architecture with local LLMs. This approach allows your live systems to remain fast while a batch process systematically corrects errors, significantly improving data quality. Implement distinct models for coarse and fine-grained judgments, and prioritize fixing root causes like string hygiene over individual data points. This strategy enhances accuracy and reduces manual intervention.

Key insights

Combining local LLMs with a Lambda Architecture effectively curates image datasets, improving accuracy and user experience.

Principles

• Separate fast, approximate paths from slow, accurate ones.
• Match model size to task complexity for efficiency.
• Fix underlying rules, not just individual data points.

Method

Implement a cold path using local LLMs for text classification and vision-based image judging. Parallelize image downloads while a single GPU processes judge calls.

In practice

• Use `ollama` to run `gemma3:12b` locally for judging.
• Employ `qwen2.5vl, 3B` for fast, coarse visual checks.
• Maintain a negative cache for unfindable photos.

Topics

• LLM as Judge
• Lambda Architecture
• Image Dataset Curation
• Local LLMs
• Vision Models
• Data Quality

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Are LLMs safe? — LLM training data curation, especially via quality filters, embeds implicit biases that shape model behavior and capabilities.
• The Infrastructure Behind Making Local LLM Agents Actually Useful — Local LLM agents require deliberate infrastructure for performance and long-session stability beyond basic model serving.
• Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs — The Mask-Proof pipeline enables scalable, reproducible evaluation of LLM step-level mathematical reasoning using masked-step tasks and an LLM-based judge.
• Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention [P] — Architectural innovations like KV Sharing and mHC optimize LLM memory and latency for practical deployment.
• LLMs: A Journey Through Time and Architecture — LLM evolution prioritizes data quality, multi-stage pre-training, and architectural efficiency over sheer model size.
• Reasoning Quality Emerges Early: Data Curation for Reasoning Models — Reasoning quality in LLMs can be effectively improved by curating SFT data based on early token loss patterns, significantly boosting efficiency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by HackerNoon.