Using observability data to prevent incidents

2026-05-22 · Source: Databricks · Field: Technology & Digital — Cloud Computing & IT Infrastructure, Data Science & Analytics, Artificial Intelligence & Machine Learning · Depth: Intermediate, medium

Summary

Engineering teams are moving from reactive incident response to proactive reliability intelligence by continuously monitoring observability data to identify accumulating risk before user-facing failures. This shift requires unified telemetry access and rapid data querying, enabling leaders to make mitigation decisions in seconds rather than days. Traditional observability focuses on "what's happening now," while reliability intelligence predicts "what's likely to happen in the next 7–30 days" by analyzing trends like SLO burn rates and p99 latency. The core challenge is not a lack of data, but a data access problem, where engineering leaders cannot query complex trends across services without significant delays or analyst support. Databricks Genie addresses this by allowing natural language interrogation of metrics, logs, traces, deployment records, and incident history, enabling proactive risk management and improving R&D efficiency. The process involves centralizing telemetry, defining leading indicators, and enabling self-service querying.

Key takeaway

For VPs of Engineering or Directors of AI/ML managing high-scale systems, your focus should shift from improving Mean Time To Resolve (MTTR) to reducing incident frequency through proactive reliability intelligence. Implement self-service querying for telemetry data to identify accumulating risk trends, such as SLO burn rate trajectory or p99 latency increases, before they manifest as user-facing incidents. This approach protects R&D capacity from emergency response and strengthens customer trust, transforming reliability management into a strategic business function.

Key insights

Proactive reliability intelligence, enabled by rapid data access, prevents incidents by identifying risk trends before they manifest.

Principles

• Reliability is an R&D efficiency and revenue protection problem.
• Reactive incident response is often an architectural data access problem.
• Leading indicators predict future incidents; lagging indicators measure past response.

Method

Centralize telemetry, define leading indicators (SLO burn rate, p99 latency trend), and enable self-service natural language querying for engineering leaders.

In practice

• Query p99 latency increases >20% over 14 days.
• Track SLO burn rate trajectory and error budget remaining.
• Integrate infrastructure cost data with performance data.

Topics

• Reliability Intelligence
• Observability Data
• Incident Prevention
• Databricks Genie
• SRE Practices
• DORA Metrics

Best for: DevOps Engineer, Director of AI/ML, VP of Engineering/Data

Related on AIssential

• From Observability to Predictive Resilience: How AI-Driven SRE Is Redefining Cloud Operations — AI-driven predictive resilience transforms SRE from reactive observability to proactive incident prevention and automated recovery.
• How Aging Data Becomes a Crisis — Proactive data aging analysis prevents GDPR non-compliance and enables informed organizational decision-making.
• You Can’t Debug What You Can’t See: Observability for Production GenAI Systems — GenAI observability must focus on understanding probabilistic system behavior, not just deterministic system state.
• Self-Healing Data Observability in Converged Architectures — Self-healing data platforms require an "Anticipate, Remediate, Immunize" (ARI) loop for autonomous data quality.
• Build an agentic incident triage assistant with Amazon Quick and New Relic — An AI agent can automate incident triage by integrating observability and task management tools into a single conversational workflow.
• Telemetry that matters: Designing sustainable, high-impact observability pipelines — Over-collection of telemetry data creates significant waste and hinders effective incident response in complex cloud-native systems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Databricks.