From Stack Traces to Seconds: Building an Agentic Triage Workflow for Data Pipelines

2026-04-11 · Source: Data Engineering on Medium · Field: Technology & Digital — Data Science & Analytics, Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, medium

Summary

An agentic triage workflow has been developed to automate the debugging and resolution of failed data pipelines in Azure Data Factory (ADF) and Databricks. This workflow addresses common issues like schema drift, which typically lead to time-consuming manual log analysis and increased Mean Time To Recovery (MTTR). The solution involves extracting raw job output JSON from Databricks using the `databricks jobs get-run-output` CLI command and piping it directly to Anthropic's Claude Code. Claude Code, prompted as an expert Data Engineer, analyzes the telemetry, identifies the root cause (e.g., `AnalysisException` due to a column name change), and generates the exact PySpark code fix, such as using the `.alias()` method. This process reduces incident resolution from minutes or hours to mere seconds, ensuring data SLAs are met and minimizing cluster compute waste.

Key takeaway

For Data Engineers managing Azure Data Factory and Databricks pipelines, integrating an AI agent like Claude Code into your incident response workflow can drastically cut down Mean Time To Recovery. Instead of manually sifting through stack traces, you should pipe raw job telemetry directly to an LLM to instantly diagnose errors and generate code fixes. This shifts your focus from tedious debugging to architecting resilient, automated operational responses, ensuring data SLAs are consistently met and reducing costly downtime.

Key insights

Automating data pipeline error triage with AI agents significantly reduces MTTR and manual debugging effort.

Principles

• Pipe raw telemetry directly to AI agents.
• Strictly prompt AI for specific outputs.
• Automate low-value, repetitive tasks.

Method

Extract Databricks job output JSON via CLI, pipe it to an LLM (Claude Code) with a precise prompt to diagnose the error and generate corrected PySpark code, then apply the fix.

In practice

• Use `databricks jobs get-run-output` for telemetry.
• Employ `claude -p "..."` for one-shot prompting.
• Apply `.alias()` for schema drift fixes.

Topics

• Agentic Triage Workflow
• Data Pipelines
• Databricks CLI
• Anthropic Claude Code
• Schema Drift

Best for: Data Engineer, MLOps Engineer, AI Engineer

Related on AIssential

• A harness for every task: dynamic workflows in Claude Code — Claude Code's dynamic workflows enable on-the-fly multi-agent orchestration to overcome single-context AI limitations.
• Build an agentic incident triage assistant with Amazon Quick and New Relic — An AI agent can automate incident triage by integrating observability and task management tools into a single conversational workflow.
• Turning Andrej Karpathy’s LLM Coding Thoughts into Claude.md — LLMs are transforming coding workflows, shifting developers from direct coding to guiding and reviewing AI agents.
• Claude Code for Data Science Projects — Claude Code's core engineering primitives surprisingly provide a strong fit for managing messy data science workflows.
• Vibe Coding and Deploying LlamaIndex Agent Workflows with Claude Code — Claude Code can autonomously develop and deploy LlamaIndex agent workflows for specific tasks like document classification.
• Claude Code, Codex and Agentic Coding #7: Auto Mode — Auto Mode enhances agentic coding safety by selectively requiring human approval for high-risk actions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Data Engineering on Medium.