Granular Usage Attribution for dbt Pipelines with Query Tags

2026-07-01 · Source: Databricks · Field: Technology & Digital — Data Science & Analytics, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Intermediate, medium

Summary

Databricks has introduced Query Tags, now in Public Preview, for dbt pipelines, enabling granular usage attribution and performance monitoring. The dbt-databricks adapter (version 1.11+) natively supports these tags, which are recorded in system.query.history. Tags can be auto-injected, providing metadata like "@@dbt_model_name" and "@@dbt_materialized" without configuration. Users can also define custom tags at the profile level (e.g., team, cost_center, project_name, env) or model level for more specific attribution. This functionality allows data teams to perform cost attribution, debug performance, and monitor workloads using simple SQL queries. A complete open-source dbt project demonstrates end-to-end configuration and dashboarding for these tags, revealing insights like specific mart tables consuming 92% of compute time in a reference project.

Key takeaway

For MLOps Engineers or Data Engineers managing dbt pipelines on Databricks, implementing Query Tags is crucial for gaining visibility into warehouse costs and performance. You can quickly identify which models or teams consume the most resources, enabling targeted optimization efforts and accurate financial attribution. Start by configuring profile-level tags for broad coverage and leverage auto-injected tags for immediate model-level insights, then explore the provided GitHub repository for a deployable self-monitoring dashboard.

Key insights

Databricks Query Tags provide granular cost and performance attribution for dbt pipelines via auto-injected and custom metadata.

Principles

• Define organization-wide tags at the profile level.
• Tag environments consistently to separate workloads.
• Use "project_name" to distinguish dbt pipelines.

Method

Configure "query_tags" in "profiles.yml" for project-wide application or in "dbt_project.yml" or model SQL for specific models. Query "system.query.history" using map access syntax to analyze tagged data.

In practice

• Use "query_tags['@@dbt_model_name']" to identify top compute-consuming models.
• Build self-monitoring dashboards from "system.query.history" data.
• Tag metric views explicitly with a feature key for analysis.

Topics

• Databricks Query Tags
• dbt Pipelines
• Cost Attribution
• Performance Monitoring
• system.query.history
• DataOps

Code references

• databricks-solutions/dbt-query-tags
• databricks/dbt-databricks
• dbt-labs/dbt-core
• databricks-solutions/dbt-query-tags

Best for: Data Engineer, Analytics Engineer, MLOps Engineer

Related on AIssential

• Query Tags: The Context Your Warehouse Queries Have Been Missing — Query Tags provide granular business context to SQL executions, enhancing workload traceability and cost attribution.
• Introducing granular cost attribution for Amazon Bedrock — Amazon Bedrock now provides granular inference cost attribution to IAM principals for enhanced financial visibility.
• 4 YAML Files Instead of PySpark: How We Let Analysts Build Data Pipelines Without Engineers — Shifting to a declarative, SQL-centric data platform empowers analysts and drastically reduces pipeline delivery times.
• Expected Goals, End-to-End: An MLOps Pipeline for Hockey on Databricks — Production-ready MLOps infrastructure, not model complexity, ensures reliable and governed machine learning deployments.
• Databricks is no longer about tuning knobs — Databricks prioritizes abstraction and automation over granular control to serve less technical users and accelerate business value.
• Why Your Data Pipeline Is Slow: A Practical Guide to Indexes — Indexes align logical query intent with physical data access, preventing PostgreSQL from doing unnecessary work at scale.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Databricks.