I Let an AI Investigate Production. Then SigNoz Made It Explain Itself.

2026-07-26 · Source: AI on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Intermediate, long

Summary

A project named Aegis demonstrates a controlled AI-driven incident response system, integrating Codex with SigNoz for production investigation. This system, developed in a hackathon environment, addresses incidents like a worker backlog where a healthy worker drowns in 104 orders while three capacity slots remain unused. Aegis uses OpenTelemetry to emit detailed evidence spans, allowing Codex read-only access to SigNoz's MCP server to diagnose issues and propose specific, bounded actions, such as increasing worker capacity by one unit. The workflow, orchestrated by n8n, enforces deterministic policy checks, optional human approval, and post-remediation verification using fresh telemetry, ensuring that proposed changes are safe and effective before execution. This architecture emphasizes a clear boundary between AI investigation and system mutation.

Key takeaway

For DevOps or SRE teams considering AI for incident response, you should prioritize building robust guardrails over full automation. Implement a system where AI investigates with read-only access to telemetry, proposes bounded actions, and requires policy validation and verification via fresh data before any system mutation. This approach ensures safety and accountability, preventing unintended consequences while still leveraging AI for faster diagnosis.

Key insights

AI-driven incident response requires strict separation of investigation from mutation, enforced by policy and verification.

Principles

• Instrument decision boundaries, not just endpoints.
• Design expiry before designing approval buttons.
• Keep mutation actions small and boring.

Method

The Aegis workflow involves OpenTelemetry evidence emission, AI investigation via read-only access to SigNoz, structured diagnosis, n8n-orchestrated policy validation, optional human approval, bounded action execution, and fresh telemetry-based verification.

In practice

• Use OpenTelemetry spans for detailed incident evidence.
• Configure AI with read-only access to observability data.
• Implement n8n for incident workflow orchestration.

Topics

• AI Incident Response
• SigNoz
• OpenTelemetry
• Workflow Automation
• Production Safeguards
• Autonomous SRE

Code references

• SomanshuKadam/aegis-autonomous-sre
• SomanshuKadam/aegis-autonomous-sre

Best for: MLOps Engineer, AI Engineer, DevOps Engineer

Related on AIssential

• Stop Chasing Logs: Why Every Engineering Team Needs an AI Incident Commander — Modern outages stem from scattered context, not lack of monitoring; AI can automate incident investigation for faster resolution.
• Engineering an Autonomous AI Study Agent: Tracing the Agentic State Machine — Robust AI agents require deep, full-stack observability via OpenTelemetry and unified platforms like SigNoz to ensure reliability and control.
• Tracer-Cloud / opensre — OpenSRE provides an open-source framework for AI SRE agents to automate production incident investigation and response.
• You Shipped an AI Agent to Production Without Testing It. So Did I — Layered testing and evaluation gates are crucial for safe AI agent deployment in high-liability workflows.
• How to Reset Your IT Operations and Build for Intelligent Response - with Luke Rotta of Charles Schwab — Incident response slows when data outpaces human contextualization; AI-driven pattern recognition and automation can accelerate resolution.
• The AI SRE: an agent that watches other agents and fixes them before anyone gets paged — Autonomous agents can diagnose and fix production issues using telemetry within strictly defined boundaries, eliminating human paging.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI on Medium.