A Pelican Learns to Ride

2026-04-28 · Source: The Jetty Blog: Ground Truth · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, short

Summary

Jetty, a company whose logo coincidentally features a pelican, conducted an extensive benchmark using Simon Willison's "pelican riding a bicycle" visual reasoning test for frontier models. They ran the task eighteen times, including eleven prompt iterations with one model and seven model iterations with one prompt, capturing every trajectory. The results, available at pelicans.jetty.bot, showcase seven agent/model combinations and seventy-one runbooks. The platform offers three views: "The Climb" ranks agent/model combos, "Head-to-Head" provides a filmstrip viewer for parallel task unfolding, and "Runbook Diffs" allows line-by-line comparison of runbook versions. This project also highlights a "judge gap," where different LLM judges (e.g., Gemini Flash vs. Claude Sonnet) assign varying scores to the same SVG, revealing calibration differences that can invalidate leaderboards. The core insight is the value of structured, inspectable benchmark artifacts.

Key takeaway

For Machine Learning Engineers evaluating generative models, recognize that benchmark scores from LLM-as-judge systems can be inconsistent due to judge calibration differences. To ensure reliable comparisons, either lock your evaluation judge for the entire comparison lifecycle or design rubrics robust enough to withstand judge changes. Focus on analyzing model trajectories and runbook diffs, not just final scores, to understand performance evolution.

Key insights

Capturing every model iteration as a structured artifact makes benchmarks inspectable like code.

Principles

• LLM-as-judge calibration varies across models.
• Leaderboards can mislead without consistent judge calibration.
• Trajectories reveal more than final scores.

Method

Jetty ran 18 benchmark tasks, iterating prompts (11x) and models (7x), capturing all 71 runbook trajectories for analysis.

In practice

• Use "Runbook Diffs" to compare prompt edits.
• Design rubrics to be judge-agnostic.
• Lock the LLM judge for consistent comparisons.

Topics

• LLM Benchmarking
• Visual Reasoning
• Generative Models
• Evaluation Metrics
• Prompt Engineering
• Model Calibration

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Prompt Engineer

Related on AIssential

• Your LLM Judge Is Not Measuring What You Think It Is — Uncalibrated LLM judges introduce systemic biases like verbosity and position preference, requiring explicit calibration for reliable evaluation.
• Introducing BenchBench — BenchBench reveals a critical divergence between LLM "Creator" and "Solver" capabilities, with GPT 5.2 excelling at benchmark generation.
• The Anatomy of an LLM Benchmark — Effective LLM benchmarking requires continuous evolution, rigorous data quality, and dynamic evaluation methods to counter rapid model saturation.
• When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability — LLM-as-judge scores are unstable when the evaluator changes, highlighting a measurement-validity problem.
• RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator — RankJudge offers a synthetic, multi-turn, reference-grounded benchmark for LLM-as-a-judge evaluation with a strict joint correctness criterion.
• My taste and the automated benchmark disagreed almost completely — The author's subjective evaluation ("taste") diverged significantly from automated LLM benchmark results, highlighting limitations in current judging methods.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by The Jetty Blog: Ground Truth.