How Stagwell built privacy-safe ID matching on Databricks
Summary
Stagwell, a Global Marketing Services Agency, developed a privacy-safe identity matching solution on Databricks to address brands' challenges in unifying fragmented first-party data with identity providers' spines. This solution utilizes Databricks Marketplace Apps, which allow brands to install Stagwell's application directly into their own Databricks workspace, ensuring raw user data never leaves their secure environment. The system combines Databricks Clean Rooms, Unity Catalog for governance, and Jobs/Notebooks for execution, with a React and Express app layer for user experience. This approach shifts from traditional data export models to an "app runs where data lives" paradigm, significantly reducing deployment time from months to minutes and enhancing compliance. Stagwell's app integrates its Identity Spine, including behavioral and attitudinal data from The Harris Poll, Harris Quest Brand, and National Research Group, to generate privacy-safe insights and facilitate audience activation via the Stagwell Agentic Targeting System (SATS).
Key takeaway
For MLOps Engineers or Data Engineers tasked with integrating external data services, you should explore Databricks Marketplace Apps and Packaged Clean Rooms. This model allows you to securely consume advanced data capabilities like identity matching without exporting sensitive first-party data, drastically reducing compliance overhead and deployment times. Consider how this "app runs where your data lives" paradigm can accelerate your data collaboration initiatives and enhance privacy.
Key insights
Databricks Marketplace Apps enable privacy-safe identity matching by running data provider algorithms directly within a brand's secure environment.
Principles
- Data processing should occur where data resides.
- Proprietary algorithms can be distributed opaquely.
- Packaged Clean Rooms streamline secure data collaboration.
Method
Brands install the Stagwell app from Databricks Marketplace, connect first-party data, and initiate a Packaged Clean Room match. The app joins brand data with Stagwell's Identity Spine, computes match results, and delivers insights for activation.
In practice
- Distribute attribution models as Marketplace Apps.
- Offer patient cohort matching tools within hospital systems.
- Provide credit risk enrichment without data export.
Topics
- Databricks Marketplace
- Identity Matching
- Clean Rooms
- Data Collaboration
- First-Party Data
- Privacy Compliance
Best for: CTO, VP of Engineering/Data, Executive, Data Engineer, MLOps Engineer, Consultant
Related on AIssential
Editorial summary, takeaway, and curation by AIssential. Original article published by Databricks.