Runbooks + RAG: How I Gave My AI SRE Agent the Context It Was Missing

2026-07-26 · Source: HackerNoon · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Intermediate, medium

Summary

An AI SRE agent designed to diagnose incidents and draft fixes requires two distinct knowledge planes: a "live plane" for real-time data like metrics and logs, and an "organizational plane" for human-written context such as runbooks, postmortems, and architecture documents. The author initially failed by stuffing organizational knowledge into the system prompt, leading to staleness and prompt bloat. The effective solution involved treating organizational knowledge as a retrieval problem, where markdown documents from a git repository are chunked, embedded, and stored in a vector database. At incident time, the agent retrieves only relevant documents, correlating live signals with this contextual knowledge for accurate diagnoses. This approach, exemplified by an incident involving a third-party API, significantly improved the agent's ability to provide specific, actionable recommendations. Key to maintaining trustworthiness are git-managed runbooks, structured postmortems, and ensuring retrieved content serves as context, not direct command.

Key takeaway

For AI Engineers building SRE agents, recognize that production readiness hinges on providing organizational context, not just model intelligence. If your agent struggles beyond demos, implement a retrieval-augmented generation (RAG) system for runbooks and postmortems. This ensures your agent interprets live telemetry within your system's unique history, transforming it from a mere demo into a truly useful colleague. Prioritize maintaining knowledge honesty through rigorous, code-like review processes for all documentation.

Key insights

The gap between AI agent demos and production is context, best bridged by retrieval-augmented organizational knowledge.

Principles

• Separate live and organizational knowledge planes.
• Treat organizational knowledge as a retrieval problem.
• Retrieved context informs diagnosis, not direct command.

Method

Organizational knowledge (runbooks, postmortems) is stored as markdown in git, chunked, embedded, and loaded into a vector database. At incident time, the agent retrieves relevant documents based on affected services.

In practice

• Store runbooks in git, next to services.
• Publish structured postmortems to the knowledge corpus.

Topics

• AI SRE Agents
• Retrieval-Augmented Generation
• Runbook Management
• Postmortem Analysis
• Vector Databases
• Incident Response

Best for: AI Engineer, MLOps Engineer, AI Architect

Related on AIssential

• How Microsoft Ships AI Agents at Enterprise Scale — The "harness" surrounding an AI model is paramount for enterprise-scale production agents, enabling reliable action and continuous improvement.
• Day 14 of Becoming an AI Developer: RAG Explained for Developers with Practical Implementation — Retrieval-Augmented Generation (RAG) integrates multiple AI building blocks for robust modern AI applications.
• Context architecture is replacing RAG as agentic AI pushes enterprise retrieval to its limits — AI agents require real-time, machine-optimized context and memory, moving beyond human-scale retrieval systems.
• How AI, RAG, and Agents Transform Mainframe Operations — RAG and agentic AI enhance mainframe operations by providing accurate, context-specific information and automating tasks.
• Context-as-a-Service: Surfacing Cross-File Dependency Chains for LLM-Generated Developer Documentation — CaaS enhances LLM documentation agents by surfacing non-obvious cross-file dependencies, improving accuracy and efficiency.
• Article: Beyond RAG: Architecting Context-Aware AI Systems with Spring Boot — CAG enhances RAG by explicitly managing runtime context, improving enterprise AI system relevance and traceability.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by HackerNoon.