Kubernetes Cluster Upgrades & Recovery

Kubernetes Cluster Upgrades and Recovery

Kubernetes ships three minor releases a year, and each is supported for roughly fourteen months. A cluster that is not upgraded regularly falls out of support within a year and a half — no security patches, managed providers forcing upgrades, and Helm charts that no longer support your version. Yet upgrades are where many teams have their worst incidents: an API removed in the new version breaks deployments, a CNI plugin is incompatible, nodes drain faster than applications can tolerate, or a control plane upgrade fails halfway.

The safe approach is routine rather than heroic: upgrade one minor version at a time, on a schedule, with automated pre-checks, backups, and a tested way back.

TL;DR

Plan for three minor releases a year and about 14 months of upstream support per release; managed providers set their own windows.
Upgrade one minor version at a time: control plane first, then nodes, within the version skew policy.
Find removed APIs in manifests, Helm charts, and controllers before upgrading.
Check add-on compatibility: CNI, CSI drivers, ingress controllers, service meshes, operators.
Take an etcd snapshot (self-managed) and make sure workloads and state can be restored.
Protect applications with PodDisruptionBudgets and replicas; drain nodes gradually.
Control planes generally cannot be downgraded — recovery means restoring from backup or failing over to another cluster.

Quick Example

A self-managed kubeadm upgrade, one minor version:

Before any of this, take an etcd snapshot on a control plane node:

Core Concepts

Releases and Support

Upstream Kubernetes maintains the three most recent minor versions with patch releases. Managed services — EKS, GKE, AKS — publish their own calendars, often with paid extended support for older versions. Falling behind compounds: skipping versions is not supported, so a cluster three versions behind needs three sequential upgrades.

Version Skew Policy

This is why the order is fixed: API servers first, then controllers and scheduler, then nodes.

Removed and Deprecated APIs

Each release can remove beta APIs deprecated in earlier releases. Objects already stored are migrated by the API server, but manifests, Helm charts, and controllers that still use removed versions fail on the next apply. Scan before upgrading with tools such as kube-no-trouble (kubent) or Pluto, check the API server's deprecated-API metrics, and update charts and operators first.

Add-on Compatibility

Cluster add-ons often lag or lead Kubernetes versions. Before each upgrade, confirm supported versions for the CNI (Cilium, Calico), CSI drivers, ingress and gateway controllers, cert-manager, service meshes, autoscalers, and every operator you run. Upgrade add-ons to versions that support both the old and new Kubernetes versions before upgrading the cluster.

Upgrading Nodes Without Outages

PodDisruptionBudgets limit how many replicas of an application may be unavailable during voluntary disruptions; drains respect them.
Enough replicas: a single-replica Deployment cannot survive a drain without downtime, PDB or not.
Surge capacity: add new nodes before draining old ones (managed node groups do this with surge settings).
Drain in batches and watch error rates between batches; pause automatically if SLOs degrade.
Stateful workloads: drain one replica at a time and confirm replication health before continuing.

A PDB that allows zero disruptions blocks drains forever — check for them before starting.

Upgrade Strategies

Blue-green clusters are much easier when every workload is declared in Git (GitOps) and state lives in managed databases or replicated storage.

Recovering From a Failed Upgrade

Control plane: downgrades are generally unsupported. For self-managed clusters, restore etcd from the pre-upgrade snapshot onto control plane nodes running the previous version — this rolls back cluster state to snapshot time. Managed services handle control plane failures themselves but rarely offer rollback.
Nodes: replace failing nodes with nodes from the previous node image or version, which the skew policy allows.
Workloads: redeploy from Git; restore persistent data from application or volume backups. See Stateful Kubernetes Backup and Restore.
Failover: with multiple clusters, shift traffic to a healthy cluster while the broken one is rebuilt.

Practice restoring etcd snapshots and rebuilding clusters before you need to.

Best Practices

Upgrade on a Calendar

Schedule upgrades every release or two, so each is a small step rather than a multi-version emergency.

Automate Pre-Upgrade Checks

Run deprecated-API scans, add-on compatibility checks, PDB audits, and backup verification in a pipeline before every upgrade.

Upgrade Lower Environments First

Move development and staging clusters a release ahead of production and run end-to-end tests there.

Keep Clusters Reproducible

Infrastructure as code for clusters and GitOps for workloads make rebuilding a cluster a routine operation rather than a disaster.

Common Mistakes

Skipping Minor Versions

Forgetting Charts and Operators

Single-Replica Critical Services

FAQ

How often should I upgrade Kubernetes?

Often enough to stay within supported versions — in practice, at least two to three minor upgrades a year. Smaller, regular upgrades are much safer than catching up several versions at once.

Can I skip Kubernetes versions when upgrading?

No. The control plane must be upgraded one minor version at a time. Nodes can lag behind the control plane within the version skew policy, which allows upgrading nodes less often.

Can I roll back a Kubernetes upgrade?

Control plane downgrades are generally not supported. Self-managed clusters can restore an etcd snapshot taken before the upgrade on the previous version; otherwise, recovery means rebuilding or failing over to another cluster.

How do I find deprecated APIs before upgrading?

Scan manifests and live resources with tools such as kube-no-trouble or Pluto, check the API server's metrics for requests to deprecated APIs, and read the release's deprecation guide.

Do managed Kubernetes services upgrade automatically?

They upgrade control planes on their own schedules or when versions reach end of support, and offer automatic node upgrades. You remain responsible for API compatibility, add-ons, and application disruption budgets.