The AI SRE: an agent that watches other agents and fixes them before anyone gets paged

2026-07-26 · Source: AI on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Intermediate, medium

Summary

The AI SRE, developed for the Agents of SigNoz hackathon, is an autonomous agent designed to diagnose and fix production issues by watching a service via SigNoz telemetry. When SigNoz raises an alarm, the AI SRE identifies the problem from real telemetry, validates a safe fix, and applies it without human intervention. The system involves a "Worker" service, SigNoz for monitoring, a "Watcher" (the AI SRE), and a "Control Panel" with three strictly defined switches the agent can manipulate. For known failures, it uses deterministic logic, while for unknown alerts, it employs an LLM investigation agent with read-only tools to query SigNoz, reasoning about issues and logging its findings as traces. This project demonstrated self-healing capabilities for issues like database failures and cost spikes.

Key takeaway

For DevOps Engineers managing production systems, consider implementing autonomous agents like the AI SRE to reduce incident response times and human toil. By strictly limiting agent actions to a defined set of safe changes and ensuring comprehensive telemetry, you can automate routine fixes for known issues. Explore integrating LLM-powered investigation for unknown alerts, but always ensure the agent's reasoning and actions are observable within your monitoring system. This approach minimizes 2 AM pages and improves system resilience.

Key insights

Autonomous agents can diagnose and fix production issues using telemetry within strictly defined boundaries, eliminating human paging.

Principles

• Constrain agent actions to a tiny, named set.
• Observe the agent's reasoning and actions.
• Telemetry must accurately reflect system state.

Method

The AI SRE (Watcher) receives SigNoz alerts, diagnoses problems from traces, checks fix safety, applies changes via a Control Panel, and records incidents, enabling self-recovery.

In practice

• Instrument services with OpenTelemetry for traces.
• Attach critical values as span attributes.

Topics

• AI SRE
• Autonomous Agents
• Observability
• SigNoz
• OpenTelemetry
• Incident Automation

Best for: MLOps Engineer, AI Engineer, DevOps Engineer

Related on AIssential

• Sandboxing, Agent Harnesses, and Agent Teamwork — Autonomous AI SREs gain value by learning and compounding operational memory across an organization, not just by accelerating incident triage.
• AURA and Open-Source Agents for Production Operations — Aura provides a declarative, Kubernetes-inspired framework for SRE agents, automating production operations with transparency and structured memory.
• Agentic DevOps at AWS — AI agents apply reasoning and tool-calling to automate DevOps pain points, enhancing operational velocity and safety.
• Tracer-Cloud / opensre — OpenSRE provides an open-source framework for AI SRE agents to automate production incident investigation and response.
• Presentation: DevOps Modernization: AI Agents, Intelligent Observability and Automation — AI agents are transforming DevOps by enabling predictive operations, intelligent rollouts, and machine-speed incident remediation.
• SRE AI Agent Safe Failure Implementation — Trust in SRE AI agents is an engineered outcome requiring grounded telemetry, safety boundaries, and progressive autonomy.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI on Medium.