I gave my agent an X-ray and hit the one SigNoz gotcha nobody warns you about

2026-07-19 · Source: AI on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Intermediate, quick

Summary

The article details instrumenting a small AI research agent, built on Nebius AI Studio and speaking the OpenAI API, with OpenTelemetry and a self-hosted SigNoz v0.133.0. The goal was to diagnose performance bottlenecks and token consumption across its three steps: "plan" (model call for sub-queries), "fetch_context" (tool call stand-in), and "synthesize" (second model call for final answer). The author, using a MacBook, Docker, and Python 3.11, encountered an unadvertised issue during SigNoz setup, which caused a half-hour delay. The article provides a walkthrough to replicate the instrumentation and avoid this specific trap, enabling detailed tracing of agent operations.

Key takeaway

For AI Engineers building multi-step agents, instrumenting your systems with OpenTelemetry and a self-hosted SigNoz is critical for diagnosing performance issues and understanding token costs. This approach allows you to pinpoint slow steps and inefficient model calls, preventing wasted development time on unobservable systems. Ensure you follow a robust setup guide to avoid common configuration pitfalls, especially with SigNoz v0.133.0, to gain immediate visibility into agent execution.

Key insights

Instrumenting AI agents with OpenTelemetry and SigNoz reveals performance bottlenecks and token usage, despite setup challenges.

Principles

• Observability is crucial for AI agent performance.
• Distributed tracing identifies slow agent steps.
• Token usage tracking aids cost optimization.

Method

Wire up an AI agent to OpenTelemetry and a self-hosted SigNoz to trace execution, identify slow steps, and monitor token consumption.

In practice

• Use OpenTelemetry for agent instrumentation.
• Deploy SigNoz for distributed tracing.
• Monitor token counts per agent step.

Topics

• OpenTelemetry
• SigNoz
• AI Agents
• Distributed Tracing
• Observability
• Performance Monitoring

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• If You Can’t Observe Your Agents, You Don’t Own Them: A Deep Dive into Multi-Agent Observability… — Agent observability, enabled by OpenTelemetry and SigNoz, is crucial for debugging, cost management, and reliability of production AI agents.
• Tracking Every Token: Granular Cost and Usage Metrics for Microsoft Foundry Agents — Granular AI agent cost and usage tracking is achievable by integrating Azure API Management and Application Insights.
• Agent Tracing and Observability: Log & Debug Complex AI Systems — Multi-agent systems require specialized observability beyond traditional logging to debug coordination and self-modification failures.
• AI Observability Starter Kit for Microsoft Foundry agents — Production-grade AI observability requires instrumented traces, automated quality and safety evaluations, and proactive alerting.
• OpenTelemetry Graduates to CNCF's Highest Maturity Level — OpenTelemetry's graduation establishes it as the de facto vendor-neutral standard for unified telemetry collection across cloud-native and AI systems.
• Network boundary for AI agents using NGINX and OpenTelemetry — Enforce and observe AI agent network access using existing cloud-native tools like NGINX and OpenTelemetry.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI on Medium.