Cilium: eBPF Networking, Network Policy & Hubble

Q: How do I find out why traffic is blocked?

Use Hubble: hubble observe --verdict DROPPED filtered by namespace or labels shows each dropped flow with the reason, including the policy decision. That is usually faster than reading policies by hand.

Cilium

Cilium is a Kubernetes networking, security, and observability layer built on eBPF — small programs loaded into the Linux kernel that process packets and system events without changing kernel code. As a Container Network Interface (CNI) plugin, it gives Pods their network connectivity; beyond that it can replace kube-proxy for Service load balancing, enforce network policies up to the HTTP and DNS layers, encrypt traffic between nodes, connect multiple clusters, and show every flow through its Hubble observability layer.

Cilium is a graduated CNCF project and the default networking for several managed Kubernetes offerings. Its eBPF datapath avoids the long iptables rule chains that slow traditional setups at scale, and its identity-based policy model keeps rules stable as Pods come and go.

TL;DR

Cilium is an eBPF-based CNI: Pod networking, Service load balancing, policy, and visibility in one component.
Policies select workloads by labels (identities), not IP addresses, and can filter HTTP paths, methods, and DNS names.
Hubble shows live flows and policy verdicts — the fastest way to see why traffic is dropped.
It can replace kube-proxy, encrypt node-to-node traffic with WireGuard or IPsec, and join clusters with Cluster Mesh.
Validate installs and upgrades with cilium status and cilium connectivity test, and upgrade one minor version at a time.
Start policies in a namespace with default deny only after observing real traffic.

Quick Example

Allow the frontend to call only GET /api/... on the backend, and nothing else:

A POST /api/orders from the frontend, or any request from another Pod, now appears in Hubble as dropped — with the policy verdict that explains why.

Core Concepts

eBPF Datapath

eBPF programs attached to network interfaces and sockets make forwarding, load-balancing, and policy decisions in the kernel. Compared with iptables-based networking, rule lookup does not grow linearly with the number of Services and policies, and Cilium can observe traffic without extra sidecars. eBPF needs a reasonably recent Linux kernel; check Cilium's system requirements for your node images.

Identities

Cilium assigns each set of Pod labels a numeric security identity and enforces policy on identities rather than IP addresses. When Pods scale or restart and get new IPs, policies keep working without rewriting rules.

Policy Types

DNS-aware egress rules (toFQDNs) allow traffic to named external services — api.stripe.com rather than a list of IPs that changes.

kube-proxy Replacement

Cilium can implement Kubernetes Services in eBPF, replacing kube-proxy. This reduces latency and rule-update cost on large clusters and enables features such as preserving client source IPs. Enabling it on an existing cluster is a migration: plan it, test it, and remove kube-proxy deliberately.

Hubble

Hubble records flows with source and destination identities, ports, HTTP or DNS details, and policy verdicts. The CLI, UI, and metrics answer the most common questions in Kubernetes networking: what is talking to what, and what is being dropped. See Kubernetes Networking.

Installing and Operating

For production, install with the Helm chart from your GitOps repository so values are versioned and reviewed. Choose the IP address management (IPAM) mode — cluster-pool, Kubernetes host-scope, or the cloud provider's mode on AWS, Azure, and GKE — before creating the cluster; changing it later is disruptive.

Upgrades

Read the upgrade notes for every minor version you cross, and move one minor version at a time.
Run the documented pre-flight check, which pre-pulls images and validates the configuration.
Upgrade a staging cluster with representative policies first, then run cilium connectivity test and watch Hubble drop rates after each step.

Troubleshooting Checklist

Best Practices

Observe Before Enforcing

Turn on Hubble and look at real flows for a namespace before writing a default-deny policy. Policies written from observed traffic rarely break production on day one.

Layer Policies

Start with a cluster-wide baseline — deny by default for application namespaces, allow DNS and essential platform traffic — then add per-application allow rules owned by the team that runs each service.

Prefer Labels and Names Over IPs

Select workloads by labels and external services by DNS names. CIDR rules are for genuinely static ranges such as corporate networks.

Treat Cilium as Critical Infrastructure

It sits on every packet. Pin versions, upgrade deliberately, monitor agent health, and alert on rising policy drops after changes. See Container Security.

Common Mistakes

Default Deny Without DNS

L7 Rules Without Understanding the Proxy Cost

Skipping Minor Versions on Upgrade

FAQ

What is Cilium used for?

Pod networking in Kubernetes, Service load balancing (optionally replacing kube-proxy), network policy from layer 3 up to HTTP and DNS, encryption between nodes, multi-cluster connectivity, and flow observability through Hubble.

Do I need a service mesh if I use Cilium?

Not necessarily. Cilium covers identity-based policy, encryption, L7 visibility, and some traffic management without sidecars. A full service mesh adds richer per-request routing, retries, and mutual TLS identity features. Compare your requirements with Service Mesh options before adding both.

How is Cilium different from Calico?

Both provide CNI networking and network policy. Cilium is built around eBPF with L7-aware policy and Hubble visibility; Calico supports several dataplanes, including iptables and eBPF, with strong BGP routing support. See Calico.

How do I find out why traffic is blocked?

Use Hubble: hubble observe --verdict DROPPED filtered by namespace or labels shows each dropped flow with the reason, including the policy decision. That is usually faster than reading policies by hand.

Does Cilium work on managed Kubernetes?

Yes. Several managed services offer Cilium-based networking directly, and Cilium can be installed on others with the provider-specific IPAM mode. Check your provider's documentation for supported modes.