Balinder Walia
A practical guide to pairing Kubeflow (train / evaluate / register) with Argo CD (GitOps deliver / rollback). For the full architecture, manifests, and anti-patterns, read the long article.
InferenceService, pipeline definition, and platform component is reconciled from Git. Train and evaluate in Kubeflow; promote by updating a Git manifest; roll back with git revert. Keep models in object storage — Git stores URIs and config, not weights.
Why both tools?
Kubeflow answers: how do we run reproducible training and evaluation on GPUs? Argo CD answers: how do we ship the resulting model to production without snowflake clusters? Using only Kubeflow often ends in manual kubectl apply for serving. Using only Argo CD leaves you without a first-class ML pipeline and experiment system. Together they form a closed MLOps loop.
The division of labour
| Concern | Tool | What lives in Git |
|---|---|---|
| Pipeline DAGs, TrainJobs, experiments | Kubeflow Pipelines / Trainer | Pipeline YAML / component specs |
| Model artifacts & lineage | Model Registry + object storage | URI + metadata — not the weights file |
| Serving, canaries, rollbacks | KServe + Argo CD | InferenceService overlays |
| Platform install & upgrades | Argo CD Applications | Helm/Kustomize for Kubeflow itself |
End-to-end loop (what good looks like)
- Data scientist merges a pipeline change → Argo CD syncs pipeline CRDs.
- Kubeflow runs training on GPU nodes (optionally queued via Kueue).
- Evaluation gates pass → model URI registered; a PR updates the prod
InferenceServicestorageUri. - Argo CD detects OutOfSync and rolls out (canary traffic first).
- Drift or bad metrics →
git revertrestores the previous URI; Argo CD reconciles.
Five practices that save you pain
- Separate repos or folders: platform (Kubeflow install) vs apps (pipelines + InferenceServices).
- Never commit model binaries — reference
s3:///gs:/// PVC URIs. - Sync waves: MySQL/MinIO (or managed equivalents) before KFP components.
- Resource limits on every step — training jobs without limits will starve the cluster.
- Promotion = PR, not a button in a notebook. Audit trail comes free.
When this stack is overkill
A single Mac Studio running Ollama for an SME chatbot does not need Kubeflow + Argo CD. Reach for this pattern when you have multiple models, GPU contention, compliance needs, or more than one environment (dev/staging/prod) that must stay identical.
Read the long version
The long article covers the 2026 Kubernetes MLOps map (KFP v2, Trainer, KServe, Kueue), Argo CD Application patterns, promotion gates, canary serving, GPU FinOps, and a rollout checklist. Published by Workstation.