Skill issue: Lessons from skilling up coding agents to use Langfuse - Marc Klingen, Clickhouse

2026-05-20 · Source: AI Engineer · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Intermediate, extended

Summary

Marc Klingen, a co-founder of Langfuse, shares lessons from developing and scaling coding agents, specifically focusing on integrating Langfuse into projects. He introduces "skills" as formalized shortcuts that enable agents to progressively acquire context and solve multi-domain problems, moving beyond rigid workflows. Langfuse created a skill to guide users through setting up observability and evaluations, addressing challenges like 478 pages of documentation, outdated pre-training context leading to hallucinations, and non-optimal, slow integration processes. Key learnings include the value of analyzing execution traces, providing agents with sitemaps and markdown-formatted documentation for efficient information navigation, implementing search endpoints to track agent queries and improve documentation, and establishing basic evaluation setups. The team also found that dynamic content should be referenced rather than duplicated, and auto-research, guided by precise target functions, significantly aids skill improvement.

Key takeaway

For AI Engineers developing or refining coding agents, prioritize structured information access and robust evaluation. Implement agent sitemaps and search endpoints to prevent hallucinations and ensure agents use up-to-date documentation. Focus on defining precise target functions for auto-research to guide agent optimization effectively. This approach streamlines agent development, reduces integration friction, and ensures agents deliver accurate, current solutions, scaling expert knowledge across your user base.

Key insights

Agents with formalized "skills" can solve complex, multi-domain problems more reliably than traditional workflows.

Principles

• Tracing agent execution reveals 80% of necessary detail.
• Agents need structured navigation for vast documentation.
• Dynamic content should always be referenced, not duplicated.

Method

Langfuse's skill development involved analyzing agent traces, providing sitemaps for documentation navigation, implementing a search endpoint for query-based information retrieval, and defining basic evaluation setups.

In practice

• Use agent sitemaps for structured documentation access.
• Implement search endpoints to track agent information needs.
• Define clear target functions for agent auto-research.

Topics

• Coding Agents
• Agent Skills
• Langfuse
• Observability Tracing
• AI Agent Evaluation
• Documentation Management

Best for: AI Architect, AI Engineer, MLOps Engineer, Director of AI/ML

Related on AIssential

• addyosmani / agent-skills — AI coding agents can achieve production-grade software quality by following structured, verifiable engineering workflows and best practices.
• From IDEs to AI Agents with Steve Yegge — AI is fundamentally reshaping software engineering, shifting focus from manual coding to agent orchestration and rapid prototyping.
• You NEED to know these vibe coding secrets — Structured agent configuration, reusable skills, and continuous loops automate advanced AI coding workflows.
• Building a Self-Improving AI Support Agent with Langfuse — Langfuse enables robust LLM production systems through comprehensive tracing, metrics, and user feedback mechanisms.
• SWE-Explore: Benchmarking How Coding Agents Explore Repositories — SWE-Explore benchmarks coding agents' repository exploration, revealing agentic methods surpass classical retrieval in line-level code localization.
• Seeing like an agent: how we design tools in Claude Code — Effective agent tool design requires understanding the model's perspective and adapting tools as its capabilities evolve.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI Engineer.