Announcing AI Driven Data Engineering

2026-03-04 · Source: Dagster Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Software Development & Engineering · Depth: Intermediate, quick

Summary

Dagster University has launched a new, free eight-lesson course titled "AI-Driven Data Engineering," designed to teach data engineers how to build production-ready ELT pipelines entirely from AI prompts. The course aims to make sophisticated data platform capabilities accessible to a broader audience by leveraging AI agents. It covers choosing appropriate AI tools, guiding agents toward Dagster best practices, and developing judgment for verifying generated output. Dagster has also invested in specific tooling, including an opinionated CLI (`dg`), maintained agent skills, and structured components, to support reliable and practical AI-driven development within its framework. The course is suitable for both new Dagster users and experienced engineers exploring agentic coding workflows.

Key takeaway

For data engineers seeking to integrate AI into their development workflows or become Dagster power users, you should explore the new "AI-Driven Data Engineering" course. It provides practical guidance on structuring projects for AI agent reliability and developing critical judgment for AI-generated code, helping you build robust ELT pipelines more efficiently and align with Dagster best practices.

Key insights

AI coding agents can significantly accelerate and standardize data engineering workflows, especially within frameworks like Dagster.

Principles

• Guide AI agents toward best practices.
• Develop judgment for AI-generated output.
• Structure projects for agent reliability.

Method

Build an ELT pipeline from prompts, focusing on AI tool selection, agent guidance for best practices, and validating generated code to achieve production readiness.

In practice

• Utilize `dg` CLI for deterministic agent actions.
• Leverage maintained agent skills for aligned output.
• Employ structured components for common integrations.

Topics

• AI Coding Agents
• Data Engineering
• ELT Pipelines
• Dagster
• AI-Assisted Development

Best for: Data Engineer, AI Engineer, Software Engineer

Related on AIssential

• Dagster 1.13: Octopus's Garden — Dagster 1.13 enhances data orchestration with AI-assisted development, improved partitioned data handling, and expanded integrations.
• How AI Is Reshaping Data Engineering: A Practical Guide With Examples — AI automates complex data engineering tasks, making engineers significantly more effective through self-healing pipelines.
• Day 5 - Agentic Engineering/VIbe Coding an AI App from Zero To Production | 🔴 Live — AI agents can achieve high test coverage and implement features in complex applications.
• The AI-First Data Engineer: 10–50x Productivity and What Changes Next — Agentic AI transforms data engineering by automating entire workflows, shifting human roles to agent operation and increasing data product creation.
• Reviewable ADRs, AI by default, and weekly readings! 💡 — ADRs and "AI by default" accelerate learning and control in AI development workflows.
• When Data Stops Being Code and Starts Being Conversation (Ep. 297) — AI agents are transforming data generation from scripted pipelines to conversational negotiation, enhancing realism and accessibility.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Dagster Blog.