Data Science Meets Devops: MLOps with Jupyter, Git, & Kubernetes

2020-08-01 · Source: Hamel Husain's Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Advanced, long

Summary

Kubeflow maintainers developed a CI/CD solution for machine learning models to manage a high influx of GitHub issues, leveraging a custom Issue Label Bot. Their initial model, trained on public repositories, predicted generic labels. They then used Google AutoML to train a Kubeflow-specific model, achieving an average precision of 72% and recall of 50% for specific labels like "area-jupyter" (0.9 precision, 0.7 recall) and "area-katib" (0.8 precision, 1.0 recall). The CI/CD pipeline, illustrated in Figure 2, uses independent controllers instead of a Directed Acyclic Graph (DAG) for preprocessing, training, validation, and deployment. This system integrates Jupyter notebooks for model development, GitOps for CI/CD, and Kubernetes with managed cloud services for infrastructure, ensuring continuous retraining and deployment of the issue labeling model.

Key takeaway

For MLOps Engineers building resilient and scalable ML CI/CD pipelines, consider adopting a reconciler-based, GitOps approach over traditional DAGs. This method, exemplified by Kubeflow's issue bot, enhances system resilience and team autonomy by allowing independent tool choices and declarative state management. You can start by adapting the provided Dockerfile and kustomize package to deploy your own ModelSync controller and defining custom lambdas and Tekton pipelines.

Key insights

Independent, reconciler-based controllers offer a resilient alternative to DAGs for ML CI/CD.

Principles

• Reconcilers ensure systems return to desired states.
• GitOps uses source control as configuration truth.
• Decentralized controllers enhance team autonomy.

Method

The solution uses two independent Kubernetes controllers: a Trainer that periodically checks model freshness and initiates retraining, and a Deployer that ensures the correct model is deployed, both driven by GitOps.

In practice

• Use Jupyter for model development.
• Implement Tekton tasks for CLI-based workflows.
• Deploy custom controllers with Kubebuilder.

Topics

• Kubeflow
• MLOps
• Kubernetes Controllers
• GitOps
• Issue Labeling

Code references

• marketplace/issue-label-bot
• kubeflow/code-intelligence
• kubernetes/git-sync
• nteract/papermill
• cli/cli

Best for: MLOps Engineer, DevOps Engineer, Machine Learning Engineer

Related on AIssential

• How Spacelift blends AI experimentation with GitOps governance — AI's rapid adoption necessitates a dual-path infrastructure approach balancing rapid experimentation with production-grade GitOps governance.
• MLOps vs DevOps: A Practical Guide for Data Scientists and IT Teams — MLOps extends DevOps to manage the unique lifecycle of ML models, data, and continuous training.
• GitOps for Low-Code: Engineering a Production-Grade CI/CD Pipeline for n8n on Kubernetes — Low-code platforms such as n8n are increasingly vital for automating workflows and integrating APIs, but their ad-hoc deployment methods prove unsustainable for mission-critical production environments.
• From GitOps to AgenticOps: Giving Kubernetes a Brain — AgenticOps layers AI reasoning onto GitOps and Kubernetes for adaptive, goal-driven operational intelligence.
• Unlocking the Full Value of Your Databricks — Databricks consolidation requires a coordination layer like Dagster for full operational visibility and governance across the entire data stack.
• GitHub availability report: May 2026 — GitHub prioritizes availability and capacity by re-architecting its infrastructure to support rapid AI-driven traffic growth.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Hamel Husain's Blog.