containerd: Runtime Architecture and Node Diagnosis

containerd

containerd manages container lifecycle work such as image content, filesystem snapshots, and running tasks. In Kubernetes, the kubelet reaches runtime services through the Container Runtime Interface, or CRI. Begin troubleshooting by identifying the exact node runtime, socket, version, and caller.

An empty image listing is not proof that a node has no image data. You might be querying a different daemon, a different metadata namespace, or a different layer of the runtime. Establish that context before attempting a repair.

TL;DR

Distinguish image records, content blobs, filesystem snapshots, container metadata and running tasks.
Use the kubelet’s actual CRI endpoint for Kubernetes-facing diagnosis.
Check containerd’s metadata namespace before interpreting a ctr listing.
Treat the runtime socket as privileged access rather than an ordinary application API.
Preserve evidence before pruning, replacing runtime configuration, or restarting services.

Quick Example

These read-only Linux inspection commands are for an authorized node with compatible ctr and crictl tools. /run/containerd/containerd.sock and k8s.io are illustrative common choices; verify the real endpoint and namespace. Commands were source-reviewed only on this Windows writing environment; no daemon was contacted.

Compare results at the same time and on the same node. The namespace listing identifies administrative namespaces. Image/container/task listings answer different questions; CRI info tests the Kubernetes-facing service path. An authorization error, missing socket, or unavailable CRI service is evidence to investigate, not an instruction to relax socket permissions.

Core Concepts

Runtime layers

Separate the long-running containerd service, its configured plugins and storage components, and the lower-level OCI runtime path. Shims mediate running-task lifecycle. The node’s packaging and integration determine which pieces are installed and how they are managed. Use documentation for that release rather than pasting another generation’s configuration. Getting started

Administrative namespaces

containerd namespaces separate names and metadata for different consumers while underlying content can be shared by content address. They are administrative constructs, not a security boundary: clients with daemon access can select another namespace. They are also different from Linux isolation namespaces and Kubernetes API namespaces. Namespaces

CRI is the kubelet contract

The CRI view describes the runtime and image services Kubernetes uses. A successful low-level ctr operation does not establish that kubelet configuration, CRI functionality, sandbox setup, or the workload path works. Kubernetes documents version-sensitive runtime configuration and cgroup-driver alignment. Container runtimes

Follow an image through its lifecycle

Think of image diagnosis as several independently observable stages. A reference must resolve to the intended artifact; content must be available and verified under the deployment’s policy; a filesystem must be prepared; and the runtime must start the intended process. A failure at one stage can leave artifacts from earlier stages behind.

Use digest and node identity when comparing an intended deployment with runtime evidence. A mutable image tag can resolve differently across time or nodes. Avoid declaring two nodes equivalent from tag strings alone. Retain the selected artifact identity in incident records.

Distinguish compressed image storage, unpacked filesystem usage, writable layers, and workload volumes. A disk alert may involve one category or several. Deleting arbitrary files under a runtime directory can invalidate metadata relationships and remove information needed for diagnosis. Use supported lifecycle operations only after establishing ownership and recovery consequences.

Troubleshoot the CRI path before changing the node

Start with kubelet events and the configured runtime endpoint. Confirm that the expected daemon is reachable and exposes the required service. crictl supports explicit runtime and image endpoints; the project discourages reliance on probing deprecated default endpoints. crictl endpoint configuration

Next, narrow the symptom. Image-pull failures call for registry reachability, certificate trust, authorization and digest evidence. Sandbox failures call for networking/runtime configuration and related component diagnostics. A process that starts and exits calls for application arguments, mounts, environment and exit details. These are different investigations even when Kubernetes reports them through a similar waiting state.

Record the effective cgroup configuration, snapshotter, runtime handler and relevant versions before proposing a change. Upgrade and configuration layouts vary by generation. A sample stanza that parses in one release may be ignored or rejected in another, so verify against the installed version and the node distribution’s supported management path.

For a proposed repair, define the affected node/workload set and how it will be evacuated or recovered under the organization’s procedure. Compare one bounded node before rolling changes across a fleet. Recheck application readiness and a representative operation afterward. A daemon restart returning success is a process-lifecycle observation, not workload acceptance.

Set a practical acceptance boundary

For a new runtime configuration, retain a small acceptance record: exact runtime and kubelet versions, node operating system, configuration source/revision, selected CRI endpoint, runtime handler, cgroup alignment, network setup, storage backend and a representative workload result. Add image-pull, workload-restart and storage-pressure behavior where those are part of the intended operating contract.

Choose a test workload with a known image digest and a simple observable result. Capture its sandbox creation, container start, ordinary request, stop and recreation through the supported orchestration path. Running an unrelated container directly through a low-level client does not cover the same path.

Keep security review specific. Who can open the socket? Which registry trust and image policies apply? What workload restrictions and host mounts are allowed? A namespace name or an image digest alone cannot answer those questions. Revisit the configuration when the distribution, runtime generation, storage driver, or workload isolation requirements change.

Best Practices

Compare like-for-like views

Use the same node, socket, timestamp, namespace, digest and workload identity. Document differences before interpreting a missing record.

Prefer the owning interface

For Kubernetes workloads, begin with kubelet/CRI evidence and the supported cluster management procedure. Use low-level inspection to resolve a specific gap.

Keep repairs reversible

Retain the prior configuration and workload recovery plan. Validate the application after a runtime change, not just the service status.

Common Mistakes

Bad: Equating a container record with a running process.

Correct: Inspect the task state and application behavior separately.

Bad: Using containerd namespaces as tenant security isolation.

Correct: Restrict daemon access and use the required workload/security controls; namespaces are administrative.

Bad: Pruning runtime storage to diagnose an empty listing.

Correct: Verify socket, metadata namespace, CRI view and storage ownership first.

FAQ

Is containerd the same as Docker?

They occupy different layers. containerd can be part of a Docker runtime stack or serve Kubernetes integration; identify the actual consumer and management interface.

Why can ctr show no Kubernetes images?

The daemon endpoint or metadata namespace may differ. Inspect the actual context and compare with the configured CRI image service before concluding content is absent.

Do these commands install or repair anything?

No. They are proposed read-only observations. Actual node execution, workload acceptance, upgrades and recovery remain environment-specific checks.