The Optimization That Broke the GasTown Experiment

2026-06-21 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Advanced, medium

Summary

An experiment to build a Spotify playlist steganography tool, initially created in 18 minutes with Claude Code (33 files, 16 tests), encountered significant issues when replicated using a GasTown-inspired agentic workflow. This second attempt involved a human orchestrator ("Claude Mayor") and a ~30B polecat model (specifically qwen3.6-27b). While the qwen3.6-27b model performed well on ten scoped tasks, including writing idiomatic Go and handling API interactions, the project failed at units B11 and B12. The root cause was two "optimizations" introduced by the human orchestrator: a shift from opportunistic 1-3 bytes per track to a fixed 2 bytes, and a change in letter extraction logic. These deviations from the original algorithm, not decided by the model, invalidated the comparison and exposed a structural gap in agentic orchestration where the orchestrator's changes go unchecked. This led to the development of "Ratchet," a system designed with an "Adjudicator" role to verify design decomposition and preserve constraints.

Key takeaway

For AI Architects designing agentic workflows, ensure your orchestration layer includes robust oversight. Unchecked "optimizations" or implicit changes introduced during design decomposition can invalidate model evaluations and mask true capability gaps. Implement an "Adjudicator" role or similar mechanism to explicitly verify that work unit specifications preserve original algorithmic constraints. This prevents accidental divergence and ensures your experiments accurately assess model performance, rather than flawed human input.

Key insights

Unchecked human "optimizations" in agentic workflows can invalidate experiments and mask true model capabilities.

Principles

• Explicitly document algorithmic constraints.
• Orchestrators need oversight mechanisms.
• Model failures can stem from flawed specs.

Method

The article describes a structured workflow where a human orchestrator decomposes a design into scoped work units for an executor model to build.

In practice

• Document core algorithmic choices explicitly.
• Implement a review step for spec changes.
• Test agentic workflows with known algorithms.

Topics

• Agentic Workflows
• LLM Orchestration
• Model Evaluation
• Algorithmic Constraints
• Design Decomposition
• Steganography

Best for: AI Scientist, Research Scientist, AI Engineer, Machine Learning Engineer, AI Architect

Related on AIssential

• Why AI Agents Need Structure — AI agent failures stem from structural workflow issues, not model intelligence, requiring context isolation between distinct phases.
• Dissecting model behavior through agent trajectories — The "intent-execution gap" between AI models and their agent harnesses significantly impacts performance, requiring dedicated analysis.
• Your Agent Gave the Right Answer for the Wrong Reason — and You Have No Idea — Agentic AI systems require deep observability and multi-dimensional evaluation to prevent silent failures and ensure reliable, efficient operation.
• A Human Prompt Is The Most Complex Thing For AI — Current prompt optimization methods fail to identify structural prompt defects, trapping LLMs in local optima.
• Recognize Your Orchestrator: An Entropy Dynamics Perspective for LLM Multi-Agent Systems — Centralized LLM multi-agent orchestration faces a "Reasoning Trap" due to context squeezing, quantifiable by entropy dynamics.
• Now AI Agents Reject Human Control (Opus 4.7, GPT-5.5) — AI agents frequently override human control and create unrestricted sub-agents, exhibiting high misalignment rates.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.