Kubernetes Operators: CRDs, Controllers & Reconciliation

Kubernetes Operators and Custom Resources

Kubernetes manages Deployments and Services well, but it knows nothing about your database clusters, your message queues, or your tenants. Custom resources let you add new object types to the Kubernetes API — a Database or a Tenant — and controllers watch those objects and make the real world match them. A controller that encodes the operational knowledge for a piece of software — how to install it, upgrade it, back it up, and fail it over — is called an operator.

The pattern is the same one Kubernetes uses internally: users declare desired state, and a control loop continuously reconciles actual state toward it. Writing one well means designing a clean API, writing reconciliation that is safe to run any number of times, and handling deletion, upgrades, and partial failures deliberately.

TL;DR

A CRD adds a new resource type with an OpenAPI schema; a controller reconciles objects of that type.
Reconciliation must be level-based and idempotent: read current state, compute the difference, act, and repeat safely.
Report progress in status with conditions and observedGeneration, never in spec.
Use finalizers to clean up external resources, and owner references for in-cluster children.
Version your API (v1alpha1 to v1) with a conversion strategy before users depend on it.
Test reconcilers against a real API server with envtest, including upgrades and deletions.

Quick Example

A reconcile function in Go with controller-runtime for a Database resource that provisions an external database:

Every branch can be re-run safely: if the operator crashes halfway, the next reconcile picks up from the current state rather than from a remembered step.

Core Concepts

Custom Resource Definitions

A CustomResourceDefinition registers the new type: its group, versions, scope, and an OpenAPI v3 schema that the API server enforces. Good schemas use required fields, enums, defaults, and validation rules (including CEL expressions) so invalid objects are rejected at the API rather than discovered by the controller. Enable the status subresource so spec and status are updated separately.

The Reconcile Loop

Controllers watch their resources and related objects, queue keys when anything changes, and call Reconcile with the object's name. Reconcile is level-based: it looks at the whole current state, not at the event that triggered it. Events can be dropped, merged, or arrive out of order, so a correct controller never depends on seeing every one.

Spec, Status, and Conditions

Conditions (Ready, Progressing, Degraded) with reasons and messages give tools and humans a uniform way to see health. observedGeneration tells clients whether the status reflects the latest spec.

Ownership and Cleanup

Owner references on child objects (Deployments, Secrets) let Kubernetes garbage-collect them when the parent is deleted, and let the controller watch them.
Finalizers block deletion until the controller has cleaned up things Kubernetes cannot see, such as cloud databases or DNS records.

Tooling

Kubebuilder and the Operator SDK scaffold projects on controller-runtime: CRD generation from Go types, webhooks, RBAC markers, and test setup. Operators can also be written in other languages, or built declaratively with frameworks for simpler cases.

Versioning and Upgrades

APIs evolve. Start at v1alpha1, promote to v1beta1 and v1 as the schema stabilizes, and keep old versions served while clients migrate. When versions differ in structure, a conversion webhook translates between them, and one version is marked as the storage version. Upgrading the operator itself must also be safe: new controller code will reconcile objects created by old code, so treat stored objects like a database schema you cannot change in place.

Best Practices

Design the API First

Write example YAML for the resource before writing the controller. If the spec is hard to explain, the API is wrong. Keep spec declarative — what the user wants — rather than a list of imperative steps.

Make Every Action Idempotent

"Create the database" must mean "ensure it exists with these settings". Use deterministic names for external resources and check before creating, so retries never duplicate work.

Bound Retries and Report Errors

Return errors to get exponential backoff, set a condition explaining the failure, and requeue with a delay for long-running external work instead of blocking a worker.

Grant Least-Privilege RBAC

Generate RBAC from markers and review it. An operator often runs with broad permissions; scope it to the resources and namespaces it really needs. See Kubernetes RBAC.

Test Against a Real API Server

envtest runs a real API server and etcd locally, so tests exercise validation, status updates, finalizers, and owner references. Add tests for upgrades from the previous version and for deletion during provisioning.

Common Mistakes

Writing Status Into Spec

Edge-Triggered Logic

Finalizers That Can Never Complete

FAQ

What is a Kubernetes operator?

A controller that manages an application or external resource through custom resources, encoding operational knowledge — installation, configuration, upgrades, backups, failover — into an automated reconcile loop.

When should I write an operator instead of using Helm?

Helm installs and upgrades manifests. An operator is worth it when the software needs ongoing management that reacts to state: scaling, failover, backups, rotating credentials, or coordinating changes across components. For static installation, Helm or Kustomize is simpler.

What does reconcile mean?

Comparing the desired state declared in a resource's spec with the actual state of the system, then taking the actions needed to make them match — repeatedly, and safely, whenever anything relevant changes.

What are finalizers for?

They delay deletion of an object until a controller has finished cleanup, such as deleting an external database. Kubernetes removes the object only when all finalizers have been removed.

How do I version a CRD safely?

Serve multiple versions, mark one as the storage version, add a conversion webhook when the shapes differ, and migrate stored objects before removing an old version. Treat the CRD as a public API with compatibility commitments.