I Deleted 4TB of Production Data by Trusting dbt’s --full-refresh Flag

2026-03-01 · Source: Data Engineering on Medium · Field: Technology & Digital — Data Science & Analytics, Software Development & Engineering · Depth: Intermediate, quick

Summary

An analytics engineer accidentally deleted 4TB of production data, representing 18 months of historical analytics, by executing `dbt run --full-refresh --select fct_orders`. The command, intended to rebuild a specific table, replaced 847 million rows with only 124,000 rows from the day's incremental load. This incident, based on real dbt data loss events from 2022-2024, highlights a critical misunderstanding of the `--full-refresh` flag's behavior. Compounding the issue, backups had failed three weeks prior and were not validated, preventing data recovery. The event underscores the necessity of understanding destructive commands and validating backup strategies.

Key takeaway

For analytics engineers managing critical data pipelines, thoroughly understand the implications of dbt's `--full-refresh` flag and similar destructive commands. Always test such operations in a development environment first. Crucially, implement and regularly validate your backup strategy to ensure recoverability, as relying on unverified backups can turn a data loss incident into a disaster.

Key insights

Misunderstanding a command's full implications can lead to catastrophic data loss, especially without validated backups.

Principles

• Validate backup strategies regularly.
• Understand destructive command behaviors.

In practice

• Test destructive operations in development.
• Implement backup validation checks.

Topics

• dbt
• Data Loss
• Data Engineering
• Backup Strategy
• Incremental Models

Best for: Analytics Engineer, Data Engineer, MLOps Engineer

Related on AIssential

• Why Your Data Pipeline Works Perfectly… Until Real Users Arrive — Data pipelines often fail due to unexpected characteristics of real-world data, not faulty code or infrastructure.
• Google Cloud deletes Australian trading fund’s infra — Cloud providers can fail catastrophically; external backups are crucial for data integrity.
• How Meta Rebuilt Data Ingestion for Petabyte-Scale Reliability — Migrating petabyte-scale data ingestion requires staged transitions, continuous validation, and robust rollback mechanisms to ensure zero downtime and consistency.
• 3‑2‑1 Backup Rule Explained: Protect Your Data from Disaster — The 3-2-1 backup rule, augmented by immutability and testing, is critical for data resilience against diverse threats.
• why smart openclaw operators are getting more careful with updates — OpenClaw operators are advised to approach system updates with caution, treating them as critical change management rather than simple curiosity, due to documented issues like packaging regressions (e…
• The Undo Button for Big Data: An Intro to Delta Lake Time Travel — Delta Lake's transaction log enables data time travel, providing an undo button for big data operations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Data Engineering on Medium.