End-to-end lineage with DVC and Amazon SageMaker AI MLflow apps

2026-04-21 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Data Science & Analytics · Depth: Intermediate, long

Summary

This post details an MLOps workflow integrating DVC (Data Version Control), Amazon SageMaker AI, and Amazon SageMaker AI MLflow Apps to achieve full model traceability. Production ML teams often struggle to link deployed models to their exact training data and reproduce past models, a critical gap in regulated industries like healthcare and finance. The proposed solution uses DVC for versioning datasets and linking them to Git commits, SageMaker AI for scalable processing, training, and deployment, and SageMaker AI MLflow Apps for experiment tracking, model registry, and lineage. The architecture ensures every model is traceable back to its precise training data, addressing reproducibility and audit requirements. Two patterns are presented: dataset-level lineage for foundational traceability and record-level lineage, which adds individual record traceability via manifests and consent registries, crucial for compliance with data exclusion requests.

Key takeaway

For MLOps Engineers building compliant and reproducible ML systems, integrating DVC with Amazon SageMaker AI and MLflow Apps is essential. This setup allows you to precisely trace models to their training data, enabling rapid audit responses and ensuring compliance with data exclusion requests, particularly in regulated sectors. You should leverage the provided companion notebooks to implement both dataset-level and record-level lineage patterns in your AWS environment.

Key insights

Integrating DVC, SageMaker AI, and MLflow provides full ML model traceability from data to deployment.

Principles

• Separate data versioning from model lifecycle management.
• Log DVC commit hashes in MLflow for lineage bridging.
• Use content-addressable storage for efficient data versioning.

Method

Preprocess data with SageMaker Processing, version with DVC, push to S3/Git. Train with SageMaker Training, pull DVC data, log DVC commit hash and metrics to MLflow. Register model in MLflow Model Registry.

In practice

• Use S3 Object Lock for tamper-evident data storage.
• Implement record-level manifests for individual data traceability.
• Orchestrate workflows with SageMaker AI Pipelines.

Topics

• DVC (Data Version Control)
• Amazon SageMaker AI
• MLflow Apps
• ML Model Lineage
• Experiment Tracking

Code references

• aws-samples/sample-amazon-sagemaker-mlflow-dvc-lineage
• aws/sagemaker-python-sdk

Best for: MLOps Engineer, Machine Learning Engineer, AI Architect

Related on AIssential

• Streamlining generative AI development with MLflow v3.10 on Amazon SageMaker AI — MLflow 3.10 on SageMaker AI enhances generative AI development with advanced tracing, evaluation, and observability features.
• Fine-tune LLM with Databricks Unity Catalog and Amazon SageMaker AI — Integrate Databricks Unity Catalog with AWS ML services for governed LLM fine-tuning and end-to-end data lineage.
• MLflow. Stop losing your best experiments: companion notebook — The notebook provides a complete MLflow workflow for experiment tracking, model registration, and deployment.
• SageMaker Pipelines: CI/CD for ML with Terraform — The provided content indicates an error page, stating "Apologies, but something went wrong on our end." It suggests refreshing the page, checking Medium's site status, or finding other content to read.
• DagsHub now supports MLflow 3.x — MLflow 3.x shifts from run-centric to model-centric tracking, improving GenAI observability and evaluation.
• H2O MLOps Enterprise Model Registry & Hugging Face | Part 8 Integration | Part 8 — A robust model repository is essential for managing and governing a large, diverse portfolio of AI models in production.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.