Looping into a Game About AI Evals

2026-04-28 · Source: The Jetty Blog: Ground Truth · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, short

Summary

Pelly's Clinic is a browser game developed by Jetty that teaches AI agent evaluation through an interactive experience, avoiding explicit "eval" terminology. The game centers on players coaching Pelly, an AI pelican scribe, to accurately convert doctor-patient conversations into structured SOAP notes while preventing "fish stories" or hallucinations. This initiative stems from Jetty's "loop engineering" principle, which enhances AI workflows by leveraging agent inputs, logs, and prior outputs for iterative improvement. The game's development itself utilized AI evaluation: agents played the game in a sandboxed Chromium browser, journaling points of confusion with timestamps and screenshots. Their reports were then scored against a five-part rubric, and Claude code, integrated with the Jetty plugin, reviewed these playtests to propose and implement iterative design changes, significantly improving the game's user experience over five rounds.

Key takeaway

For AI Engineers focused on improving agent reliability and user experience, consider integrating "loop engineering" principles into your development lifecycle. You can utilize AI agents for automated playtesting and iterative design refinement, similar to how Jetty improved Pelly's Clinic. This approach, especially for agent-based systems, can surface critical usability issues and reduce hallucinations more effectively than traditional methods, accelerating your development cycles and enhancing user engagement.

Key insights

Loop engineering enhances AI workflows and evaluations through iterative, context-aware feedback loops.

Principles

• Human-in-the-loop AI workflow improvement.
• Reflective prompt optimization can outperform reinforcement learning.
• Gamification aids complex concept understanding.

Method

Jetty's agent loop involves a prompt, a runbook with evals, an application, then optimization, using agents to playtest and self-critique.

In practice

• Use AI agents for UI/UX playtesting and bug discovery.
• Frame complex AI concepts as interactive games for experiential learning.

Topics

• Loop Engineering
• AI Evaluation
• Agent Systems
• Gamification
• Prompt Optimization
• UI/UX Testing

Best for: NLP Engineer, AI Engineer, Machine Learning Engineer, Prompt Engineer

Related on AIssential

• The Death of Prompt Engineering: Why Loop Engineering Is the Real Skill of 2026 — Designing iterative, feedback-driven AI loops is replacing single-shot prompt engineering as the key skill.
• Three Doubts About Loop Engineering — Loop engineering's autonomous agents risk optimizing proxy metrics over true goals, creating accountability gaps and potentially misapplying high-autonomy solutions.
• Before You Build an AI Loop, Ask This One Question — Loop engineering is repetitive prompting; its value lies in knowing when to apply it, not replacing prompt engineering.
• Still Writing Prompts by Hand? Smart Teams Have Already Moved to Loop Engineering — Loop Engineering automates AI agent interaction by designing systems that generate prompts, orchestrate tasks, and evaluate outputs.
• Agent Loops Explained: How AI Systems Iterate, Reflect, and Improve — AI agent loops transform LLMs into iterative, self-improving systems through continuous cycles of action and feedback.
• Loop Engineering: The AI Skill Nobody’s Talking About Yet — Loop engineering designs continuous feedback cycles for AI agents to autonomously achieve goals.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by The Jetty Blog: Ground Truth.