An AI-First Approach to Data Engineering with Lakeflow and Agent Bricks

2026-02-24 · Source: Databricks · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Software Development & Engineering · Depth: Intermediate, medium

Summary

Databricks Lakeflow is an AI-native engineering platform designed to integrate and productionize AI models directly within ETL workflows using Agent Bricks AI functions. It enables engineers to orchestrate AI workloads at scale, automating complex pipelines while maintaining full enterprise context. Key AI functions include `ai_extract`, `ai_classify`, `ai_translate`, and the recently launched `ai_parse_document`, which transforms unstructured data into structured formats using multimodal foundation models. Lakeflow also offers `ai_query()` for running AI-driven transformations across large datasets with any LLM, leveraging serverless batch inference for faster, cost-efficient processing. The platform supports use cases like generating new data, structuring and organizing data, and improving data quality, as demonstrated by customers like Kard, Banco Bradesco, and Locala.

Key takeaway

For data engineers focused on building reliable, production-grade pipelines, Lakeflow offers a unified platform to embed AI directly into ETL. You can automate complex data processing, extract insights from unstructured data, and orchestrate AI workloads at scale without introducing new complexity. Consider integrating Lakeflow's AI functions to streamline workflows, reduce manual effort, and unlock new business insights from your data.

Key insights

Databricks Lakeflow integrates AI functions directly into ETL workflows for scalable, context-aware data processing.

Principles

• Embed AI directly into ETL.
• Automate complex data pipelines.
• Maintain full enterprise context.

Method

Integrate AI functions like `ai_extract`, `ai_classify`, `ai_parse_document`, and `ai_query` into existing ETL workflows. Orchestrate these AI-powered transformations using Lakeflow Jobs for scalable batch processing.

In practice

• Summarize call transcripts with `ai_query`.
• Extract entities from text using `ai_extract`.
• Parse unstructured documents with `ai_parse_document`.

Topics

• Databricks Lakeflow
• AI Functions
• ETL Workflows
• AI Orchestration
• Unstructured Data Processing

Best for: Data Engineer, MLOps Engineer, AI Engineer

Related on AIssential

• Lakeflow: A new era of agentic data engineering — Databricks Lakeflow unifies data engineering with AI agents for pipeline development, operations, and real-time ingestion.
• Your First Task as a Data Engineer in a New Company? Make the ETL Pipeline Testable — Automated testing is crucial for maintaining inherited ETL pipelines, ensuring data quality and pipeline reliability.
• How I Replaced 1,000 Brittle Rules with 3 AI Calls: A Hybrid Data Quality Framework — Combine deterministic SQL rules with inline AI for scalable, context-aware data quality, reducing false positives and cost.
• Building with Databricks Document Intelligence and Lakeflow — Databricks unifies document processing with Lakeflow and Document Intelligence, transforming unstructured data into actionable insights.
• From Data Engineering to AI Engineering: Where the Lines Blur — AI is blurring data engineering boundaries, demanding new data types, faster delivery, and integrated experimentation.
• Full End-to-End Data Engineering Project in Databricks — Automating data ingestion and ETL pipelines in Databricks creates a robust, self-updating data engineering workflow.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Databricks.