Building a Cluster-Aware AI Agent with Kubernetes, Argo CD, and GitOps

2026-06-25 · Source: Cloud Native Computing Foundation · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Intermediate, medium

Summary

The article describes building a self-hosted, read-only AI agent inside a Kubernetes cluster using GitOps principles. This agent leverages Ollama to serve a local Mistral 7B LLM and a FastAPI application for its API and chat UI. A CI/CD pipeline, orchestrated by GitHub Actions for image builds and Argo CD Image Updater for committing new tags to Git, ensures continuous deployment via Argo CD. The agent observes live cluster state through a dedicated read-only Kubernetes `ServiceAccount` and `ClusterRole`, enabling Retrieval-Augmented Generation (RAG) for actionable diagnostics without data leaving the cluster. This design provides auditable behavior through `git log` and demonstrates two modes: a general `/ask` endpoint and a cluster-aware `/diagnose` endpoint.

Key takeaway

For DevOps or Platform Engineers evaluating AI agent integration, this self-hosted, read-only Kubernetes agent provides a concrete starting point. By deploying a local LLM with Ollama and integrating it with live cluster state via read-only RBAC, you can safely experiment with grounded diagnostics. This approach allows you to understand the full agent loop and its GitOps-driven CI/CD without external data egress, enabling secure, auditable, and practical AI adoption within your infrastructure.

Key insights

A self-hosted, read-only AI agent within Kubernetes leverages local LLMs and GitOps for cluster-aware diagnostics, ensuring data privacy and auditable operations.

Principles

• Cluster-aware agents observe live state for grounded reasoning.
• Read-only RBAC bounds agent risks, enabling safe iteration.
• GitOps ensures auditable, single source of truth for agent behavior.

Method

Deploy Ollama and FastAPI pods with a read-only `ServiceAccount`. Use GitHub Actions for image builds, Argo CD Image Updater to commit new tags to Git, and Argo CD for cluster reconciliation.

In practice

• Serve local LLMs with Ollama (e.g., Mistral 7B).
• Implement RAG by injecting live cluster state into prompts.
• Craft system prompts to define agent personality and output.

Topics

• Kubernetes
• AI Agents
• GitOps
• Argo CD
• Ollama
• Retrieval-Augmented Generation

Code references

• MaryamTavakkoli/local-k8s-ai-agent
• argoproj-labs/argocd-image-updater

Best for: MLOps Engineer, DevOps Engineer, AI Engineer

Related on AIssential

• From GitOps to AgenticOps: Giving Kubernetes a Brain — AgenticOps layers AI reasoning onto GitOps and Kubernetes for adaptive, goal-driven operational intelligence.
• Production LiteLLM on AWS EKS: High Availability with GitOps — A robust LiteLLM deployment on AWS EKS with GitOps unifies LLM access, ensuring high availability and cost efficiency.
• Validate Kubernetes for GPU Infrastructure with Layered, Reproducible Recipes — AI Cluster Runtime simplifies Kubernetes configuration for AI workloads via validated, version-locked, and reproducible recipes.
• Build high-performance generative AI systems with Strands Agents, NVIDIA NIM, and Amazon Bedrock AgentCore — Integrating NVIDIA NIM, Bedrock AgentCore, and Strands Agents enables high-performance, scalable, and observable multi-agent generative AI systems on AWS.
• GitOps for Low-Code: Engineering a Production-Grade CI/CD Pipeline for n8n on Kubernetes — Low-code platforms such as n8n are increasingly vital for automating workflows and integrating APIs, but their ad-hoc deployment methods prove unsustainable for mission-critical production environments.
• Data Science Meets Devops: MLOps with Jupyter, Git, & Kubernetes — Independent, reconciler-based controllers offer a resilient alternative to DAGs for ML CI/CD.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Cloud Native Computing Foundation.