Why R&D Data Belongs in the Lakehouse - and Why Agents Need It There

2026-07-21 · Source: Databricks · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Cloud Computing & IT Infrastructure · Depth: Intermediate, long

Summary

cellcentric, a joint venture of Daimler Truck and Volvo Group, spent four years building a "Data Hub" on Azure and Databricks to manage R&D data for heavy-duty fuel cell systems. This platform, leveraging Unity Catalog, Lakehouse Federation, and Delta Sharing, integrates diverse enterprise sources like SAP S/4HANA and IoT telemetry into data products such as the "Fuel Cell Passport." The Data Hub provides a governed context layer for both human engineers and AI agents, exposing data through a user interface and an MCP server. A key innovation is treating "context coverage" as a first-class data quality metric, with 27 published data products averaging 90% column-comment coverage. The architecture ensures consistent governance and observability for AI agent access, integrating tools like Unity AI Gateway and MLflow tracing.

Key takeaway

For AI Engineers building industrial AI solutions, prioritizing a governed lakehouse foundation is crucial. Your agents need a context-rich Data Hub, integrating Unity Catalog, data products, and robust identity management, to ensure trustworthy and explainable outputs. Implement context coverage as a key quality metric and integrate agents into your development loop to accelerate data product delivery and improve documentation quality.

Key insights

Industrial AI agents require a governed, context-rich data lakehouse foundation to provide trustworthy, explainable answers.

Principles

• Context coverage is a first-class data quality metric for AI.
• Governance must be designed into the platform from the start.
• Documentation should be part of the engineering workflow.

Method

Build a governed data product platform on a lakehouse, integrate diverse sources, enrich with context metadata, and expose via unified interfaces for humans and agents, ensuring identity flows through for authorization.

In practice

• Implement context-coverage badges for data products.
• Use AI-assisted tools for column-level documentation.
• Route foundation model traffic through a single AI Gateway.

Topics

• Lakehouse Architecture
• Industrial AI
• Data Governance
• Unity Catalog
• Data Products
• AI Agents
• Context Coverage

Best for: AI Architect, CTO, VP of Engineering/Data, Data Engineer, AI Engineer, MLOps Engineer

Related on AIssential

• The Agentic Enterprise Just Got Real — AI's primary challenge is context management, not intelligence, requiring robust governance for agentic systems.
• The Contextual Revolution: The Rise of the Metadata Engineer in the Intelligence Economy — Metadata engineering is crucial for building trusted, context-rich data foundations essential for advanced AI and mitigating hallucinations.
• Multimodal Data Integration: Production Architectures for Healthcare AI — Multimodal data integration in healthcare requires a unified, governed lakehouse architecture to overcome production challenges.
• The Data Agent Stack - Part 2: The Data Foundation Is the Agent — The reliability of data agents hinges on a well-governed data foundation that explicitly encodes meaning, quality, and ownership for machine consumption.
• D&B's database of 642 million businesses was built for humans, not AI agents. So they rebuilt it. — Enterprise data systems built for humans require fundamental redesign for AI agents.
• Talk to all your data, wherever it lives — Databricks Lakehouse Federation unifies disparate enterprise data for AI-driven natural language querying without migration.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Databricks.