Linux Boot, Storage, and Filesystem Recovery

Locate the failing boot stage before attempting a repair. A missing root filesystem, failed secondary mount, exhausted filesystem, and damaged service configuration can look similar from a distant monitoring screen, but they require different recovery plans. Preserve the error, identify the actual device and filesystem, and establish a recoverable copy before operations that can change storage.

The useful first question is what still works. Can firmware find the boot entry? Can the kernel start? Can the early userspace find the root volume? Can the host mount the filesystems its services require? Work from the last confirmed stage toward the first failing boundary. A reboot is an experiment with consequences, not an all-purpose diagnostic.

TL;DR

Quick Example

Run these inspection commands on an authorized Linux system with the listed utilities and suitable access. Journal output can contain private operational details; inspect it locally and share only necessary excerpts.

The sequence asks which filesystem contains /var, whether blocks or inodes are exhausted, which units failed, and which errors were logged in the current boot. It does not diagnose the cause by itself. An early boot failure may prevent these commands from being available at all. No mount, repair, deletion, configuration change, or reboot was performed for this example.

Core Concepts

Boot stages define the first investigation

Systemd's boot overview describes firmware handing control to a boot loader or kernel, the kernel using an optional initramfs to locate the root filesystem, and the host system manager mounting remaining filesystems and starting services. It also notes that service initialization is substantially parallel rather than a single universal sequence. System boot process

Use that structure to interpret the evidence. If the kernel never starts, a service configuration is unlikely to be the first cause. If the root filesystem mounts and a later service fails, rewriting a boot loader is a large and poorly targeted intervention. Record the exact stage and error instead of labeling everything a boot problem.

A path and a device are different facts

Before responding to a full directory, determine which mounted filesystem contains it. Findmnt's target option resolves a path to its filesystem, which helps avoid investigating the wrong volume. Mount namespaces and rescue environments can show a different view from the application's original runtime, so record where the inspection occurs. Findmnt reference

Capacity also has multiple dimensions. Available blocks and available inodes answer different questions. A workload can create enough small files to exhaust inode capacity while substantial byte capacity remains. Conversely, a few large files can fill blocks without exhausting inodes. Treat those observations as clues to investigate; neither justifies deleting arbitrary files. Df capacity reporting

This is a diagnostic map, not a command sequence that repairs every host. The next action depends on the observed system and the consequences of losing its data.

Filesystem repair has a specific implementation

The fsck manual explains that fsck is a front end for filesystem-specific checkers. The appropriate procedure therefore depends on the actual filesystem and device state. Do not assume a command suitable for one filesystem is valid for another, or run a repair against an active mounted filesystem without the relevant documented procedure. Fsck reference

If hardware or underlying storage may be failing, repeated repair attempts can complicate recovery. Preserve the original state and involve the storage/recovery owner before making destructive changes. A successful checker exit is also not an application-level integrity test; recovered files still need validation against the workload's expectations.

Best Practices

Keep an incident record with the last known working state, recent changes, boot stage, errors, actual volume identifiers, filesystem type, and access environment. Separate observations from hypotheses. A remembered device name is weaker evidence than a verified mapping on the affected host, especially after hardware changes or recovery-media boot.

Review configuration changes against a known version. A malformed mount entry and a missing storage dependency can produce overlapping symptoms, so compare declared configuration with actual devices and ordering requirements. Restore a reviewed configuration through an authorized procedure rather than copying a generic fstab from another machine.

For capacity incidents, identify the producer and retention requirements before freeing space. Stopping uncontrolled growth may be necessary, but deleting active logs, databases, or application files can create additional failures. Choose a response with the service and data owners, and verify both recovered capacity and application behavior afterward.

Make recovery rehearsal part of normal operations. Confirm that authorized operators can access the console, obtain required encryption material through the approved process, find the correct recovery media, and restore a representative backup. A documented recovery plan is useful only if its dependencies remain available when the host does not boot.

Reverse the storage-repair hypothesis when evidence points elsewhere. If mounts and capacity are sound but a service fails after a configuration deployment, investigate that deployment. If IO errors continue after a filesystem operation, do not repeatedly treat the filesystem as the sole cause. Escalate the failing storage boundary with preserved evidence.

Common Mistakes

Assuming every emergency shell means corruption

Running a remembered repair command

Rebooting repeatedly

Declaring recovery at the login prompt

FAQ

Should I run fsck immediately?

Establish the device, filesystem, mounted state, symptoms, and preservation plan first. Fsck dispatches to filesystem-specific tools; follow the documented recovery procedure appropriate to that filesystem and situation.

Does free disk space rule out capacity problems?

No. Inspect the filesystem containing the affected path and distinguish blocks from inodes. Also confirm that your rescue or container environment is observing the same storage boundary as the failing workload.

What proves the recovery worked?

The required boot and mount stages succeed, the underlying error is addressed, and the application's data and functions pass relevant checks. Retain the procedure and evidence so the next incident starts with more than a familiar command.

Related Topics

References