Kubeflow: ML Pipelines & Training on Kubernetes

Kubeflow

Kubeflow is an open-source platform for machine learning on Kubernetes. It packages the pieces an ML team needs — interactive notebooks, reproducible pipelines, distributed training, hyperparameter tuning, a model registry, and model serving — as Kubernetes-native components that run on any conformant cluster, in the cloud or on premises. Kubeflow is a CNCF project, and its Pipelines format is also used by managed services such as Google's Vertex AI Pipelines.

Kubeflow is powerful for organizations that already operate Kubernetes well and want portable, self-hosted ML infrastructure. It is also a substantial platform to run: authentication, multi-tenancy, storage, GPUs, and upgrades all need ownership. Teams should adopt the components they need rather than the whole stack by default.

TL;DR

Kubeflow is a set of Kubernetes-native components for the ML lifecycle; you can install them together or individually.
Kubeflow Pipelines (KFP) define workflows in Python with the KFP SDK, compiled to a portable pipeline spec.
Pipeline steps are containerized components passing artifacts; runs are tracked with metadata and cached.
The Trainer runs distributed training jobs (PyTorch and others); Katib tunes hyperparameters.
KServe serves models with autoscaling; the Model Registry tracks model versions.
Multi-user isolation uses profiles (namespaces) with authentication; operating it requires real platform engineering.

Quick Example

A pipeline with one training component, written with the KFP v2 SDK:

Submitting a run to a Kubeflow Pipelines endpoint:

The compiled YAML is portable: the same file runs on self-hosted Kubeflow Pipelines or a managed KFP-compatible service.

Core Concepts

Components of the Platform

Pipelines, Components, and Artifacts

A component is a containerized function with typed inputs and outputs. Lightweight Python components (as above) package a function with a base image and dependencies; container components wrap existing images. Components exchange parameters (small values) and artifacts (datasets, models, metrics) stored in object storage. The pipeline graph, run history, and artifact lineage are recorded as metadata, and caching skips steps whose inputs have not changed.

Distributed Training and Tuning

The Trainer (formerly the Training Operator) launches multi-node, multi-GPU training jobs, configuring the processes and networking that frameworks such as PyTorch's distributed training expect. Katib runs many trials with different hyperparameters using search algorithms (random, grid, Bayesian optimization, early stopping) and records the best configuration. See PyTorch Distributed Training.

Serving and Registry

KServe deploys models as InferenceService resources with autoscaling (including scale to zero), request batching, canary rollouts, and support for many frameworks and LLM runtimes. The Model Registry links trained models to their pipelines, metrics, and versions so deployments are traceable. See PyTorch Model Deployment.

Multi-User Isolation

Kubeflow maps users and teams to profiles, each backed by a Kubernetes namespace with its own notebooks, pipelines runs, and resources. Isolation depends on the surrounding setup: an identity provider and authentication proxy, Kubernetes RBAC, network policies, resource quotas, and per-namespace credentials for storage. Treat it like any multi-tenant cluster. See Kubernetes Multi-Tenant Isolation.

Operational Requirements

Kubernetes expertise: Kubeflow installs many controllers, webhooks, and custom resources; upgrades need planning.
Authentication and ingress: an identity provider, TLS, and an authenticating gateway.
Storage: object storage for artifacts and a database for metadata, both backed up.
GPUs: node pools, drivers, and scheduling policies for training and serving.
Observability and cost: monitor pipeline failures, cluster utilization, and idle notebooks.

Kubeflow and Managed Alternatives

Best Practices

Adopt Components Incrementally

Start with the component that solves a concrete problem — often Pipelines or KServe — rather than installing everything.

Keep Components Small and Typed

Make each step do one thing with typed inputs and outputs, so steps can be cached, reused, and tested independently.

Pin Images and Dependencies

Use fixed base images and package versions in components so pipeline runs are reproducible months later.

Version Pipelines in Git

Store pipeline code and compiled specs in version control, and build and submit them from CI rather than from notebooks.

Common Mistakes

Installing the Whole Platform Without Owners

Unpinned Component Dependencies

Ignoring Notebook Costs

FAQ

What is Kubeflow used for?

Running machine learning workflows on Kubernetes: notebooks for exploration, pipelines for reproducible training and data processing, distributed training, hyperparameter tuning, model registry, and model serving.

Do I need to install all of Kubeflow?

No. Components such as Kubeflow Pipelines, Katib, the Trainer, and KServe can be installed individually, and many teams use only one or two.

What is Kubeflow Pipelines?

A component of Kubeflow for defining ML workflows in Python with the KFP SDK. Pipelines compile to a portable specification and run as containerized steps on Kubernetes, with tracked artifacts, metadata, and caching.

Is Kubeflow the same as MLflow?

No. MLflow focuses on experiment tracking, model packaging, and a model registry, and runs anywhere. Kubeflow is a Kubernetes-based platform for pipelines, training, tuning, and serving. Teams sometimes use them together.

Is Kubeflow hard to operate?

It requires solid Kubernetes skills: authentication, multi-tenancy, storage, GPUs, and upgrades all need attention. Managed services or lighter stacks may suit teams without dedicated platform engineers.