EvoClawBench: Can Agents Learn Reusable Skills from Their Own Runs?

2026-07-14 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Software Development & Engineering · Depth: Expert, extended

Summary

EvoClawBench is a new benchmark designed to evaluate whether large language model (LLM) agents can learn reusable skills from their own execution runs. It features 100 tasks and 502 sub-problems spanning coding, data, office, security, operations, and domain-document workflows, supporting multiple agent runtimes like OpenClaw and nanobot. The benchmark compares three strategies: Baseline (direct execution), PreSkill (skill authoring before execution), and PostSkill (skill summarization from first-run evidence for a fresh second execution). Experimental results show that direct Baseline performance is highly runtime-dependent, with OpenClaw below 20% and nanobot ranging from 56.45% to 96.13%. Self-authored skills yield mixed effects; for instance, nanobot GPT-5.4 maintains over 96% across modes, while nanobot DeepSeek-V4-Pro drops from 77.77% to 4.80% with PreSkill. Skill workflows also add substantial overhead, with end-to-end token-efficiency ratios between 0.21 and 0.40, indicating that skill learning is selective and cost-sensitive.

Key takeaway

For AI Scientists and Machine Learning Engineers designing or deploying LLM agents with self-authored skill capabilities, you should critically evaluate the actual benefits of skill creation. This process introduces significant overhead in terms of tokens, cost, and time, and does not guarantee performance improvements. Focus on implementing selective skill creation policies and rigorous validation before reuse to ensure that generated skills genuinely enhance agent performance and justify their associated costs, rather than assuming automatic gains.

Key insights

LLM agents' ability to learn reusable skills from their own runs is selective, runtime-dependent, and not an automatic benefit.

Principles

• Agent performance is strongly runtime-dependent.
• Self-authored skills do not guarantee monotonic gains.
• Skill creation adds substantial token, cost, and time overhead.

Method

EvoClawBench evaluates agents using Baseline (direct), PreSkill (author then execute), and PostSkill (execute, summarize evidence, then execute with skills) strategies, measuring execution-only and end-to-end metrics.

In practice

• Implement selective skill creation policies.
• Validate generated skills before reuse.
• Audit skill content, not just counts.

Topics

• LLM Agents
• Skill Learning
• Benchmark Evaluation
• Procedural Memory
• Agent Runtimes
• Cost-Efficiency

Code references

• pzy2000/EvoClawBench

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows — LLM-generated skills do not reliably improve performance for data science tasks over direct prompting.
• EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer — EvoAgentBench evaluates LLM agent self-evolution by isolating procedural knowledge transfer, moving beyond simple task accuracy or memory retention.
• New framework lets AI agents rewrite their own skills without retraining the underlying model — Memento-Skills enables AI agents to autonomously evolve their capabilities by rewriting their own executable skills.
• Formal Skill: Programmable Runtime Skills for Efficient and Accurate LLM Agents — Formal Skill transforms LLM agent capabilities from informal text prompts into token-efficient, enforceable runtime-native protocols.
• SkillCAT: Contrastive Assessment and Topology-Aware Skill Self-Evolution for LLM Agents — SkillCAT enables LLM agents to self-evolve skills by contrastive assessment and topology-aware execution, boosting performance without training.
• What Should a Skill Remember? Quality--Cost Trade-offs in Cost-Aware Skill Rewriting for Language Model Agents — Skill rewriting for LLM agents must preserve operational anchors to balance quality and total execution cost, not just shorten prompts.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.