Branching databases like code: a CI/CD pattern for Lakebase, in production at Glaspoort
Summary
Glaspoort, a Dutch fiber infrastructure company, implemented a CI/CD pattern for its Databricks Lakebase OLTP database to accelerate data team output and improve application development. Facing outdated BI reports and a sprawl of one-off analyses, Glaspoort shifted to a custom front-end application leveraging Databricks products like Genie for chat, AI/BI Dashboards for self-service, Agent Bricks for alerts, and Lakebase for transactional data. This setup merges analytical and operational environments. A critical component is their database branching strategy, treating Lakebase changes like application code via pull requests and CI. They address the challenge of testing against production-like databases by always branching long-lived environments (development, acceptance) directly from production, avoiding the "reset-from-parent trap" and ensuring environments stay fresh. Ephemeral per-PR branches are created from production, migrations are replayed, and the full test suite runs against the new app image and migrated branch before merging.
Key takeaway
For MLOps Engineers or Software Engineers managing transactional databases with CI/CD, adopting a "branch from production" strategy for long-lived environments like development and acceptance will eliminate environment drift and costly resets. You should create ephemeral database branches for each pull request, treating migrations as the schema's source of truth. This approach ensures rapid, reliable testing against production-like data, significantly increasing iteration speed and reducing "who broke acceptance?" incidents. Consider your team's velocity needs when choosing between PR stacking or per-PR promotion.
Key insights
Branching OLTP databases directly from production and treating migrations as truth enables robust, fast CI/CD.
Principles
- Long-lived database environments should branch directly from production.
- Database migrations are the authoritative source of truth for schema.
- Use ephemeral, disposable database branches for each pull request.
Method
CI creates ephemeral pr-xxxx branches from production, replays migrations, and runs full test suites against a staging app image. Merged PRs re-branch, re-replay, and re-test before applying to target branches, with promotion via manually approved gates.
In practice
- Implement copy-on-write database branching for CI/CD.
- Use "databricks postgres generate-database-credential" for short-lived tokens.
- Enforce user authorization at the application layer, not database.
Topics
- Databricks Lakebase
- Database Branching
- CI/CD Pipelines
- Postgres OLTP
- Data Governance
- Azure DevOps
Best for: MLOps Engineer, Software Engineer, Data Engineer
Related on AIssential
See Counsel's argued verdicts on the open AI decisions leaders are weighing →
Editorial summary, takeaway, and curation by AIssential. Original article published by Databricks.