Dark Factory: How OpenClaw Ships Faster Than You Can Read the Diff — Vincent Koc

2026-05-12 · Source: AI Engineer · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, long

Summary

The presentation "Malleable Evals: From Static AI Measuring to Adaptive Systems" by Vincent focuses on the evolving landscape of AI evaluation, particularly in the context of agentic AI. It highlights a critical gap between traditional static benchmarks used in AI/data science and the dynamic, adaptive nature of modern AI applications, contrasting them with software engineering practices like chaos engineering and observability. The speaker argues that current evaluation methods, which rely on handcrafted, offline tests, are insufficient for AI systems that self-optimize and adapt based on user intent. The discussion traces the evolution from prompt engineering to context engineering, and projects a future of "intent engineering" where machines self-optimize. The core proposition is to shift from static evaluations to adaptive, always-on systems that leverage telemetry and self-curated test suites to continuously monitor and correct agent behavior, addressing the "eval calcification problem" where static methods fail to keep pace with rapidly changing AI.

Key takeaway

For AI Architects and Research Scientists developing agentic systems, your evaluation strategy must move beyond static benchmarks. Embrace an "agentic mindset" for evals, treating them as self-optimizing, living agents that adapt to changing user intent and application behavior. Focus on defining desired end states and leveraging telemetry to enable agents to self-correct, ensuring your systems remain robust and aligned with dynamic operational realities.

Key insights

AI evaluations must evolve from static benchmarks to adaptive, self-optimizing systems to match dynamic agentic AI.

Principles

• AI applications are not static software.
• Evals should adapt with applications.
• Machines can self-optimize based on intent.

Method

Shift from static benchmarks to intent-based outcomes by defining ambiguity and personality, building rubrics, self-curating suites from traces, and implementing online, always-on evaluation optimizations with telemetry in the loop.

In practice

• Use traces to self-curate test suites.
• Implement always-on evaluation optimizations.
• Integrate telemetry for self-correction.

Topics

• Malleable Evals
• Adaptive AI Systems
• LLM Evaluation
• Intent Engineering
• Agentic AI

Best for: AI Architect, AI Scientist, Research Scientist, AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Is it agentic enough? Benchmarking open models on your own tooling — Agentic software optimization requires benchmarking not just task success, but also agent effort and interaction patterns across diverse models.
• Evals Skills for Coding Agents — Improving infrastructure around AI agents, especially evaluation capabilities, is more critical than solely improving the underlying model.
• The Sequence Opinion #806: The Emergence of the Agent-as-a-Judge: Why Evals Need a Reasoning Engine — AI evaluation is evolving from simple LLM-as-a-judge to agent-as-a-judge with reasoning capabilities.
• AI Agents of the Week: Papers You Should Know About — Current AI agent benchmarks overstate capabilities; real-world reliability demands trajectory-aware evaluation and specialized multi-agent architectures.
• 🥇Top AI Papers of the Week — AI advancements span self-improving agents, social intelligence, new benchmarks, automated safety, architectural innovations, and efficient agentic systems.
• The Agentic Gap: What Enterprises Think vs. What Actually Works With Jeff Dalton — Classical AI agent principles remain vital, with modern LLMs enabling new implementation methods and complex agentic systems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI Engineer.