Ubuntu Root Filesystem Space Exhaustion: A Runbook from Immediate Containment to Capacity Governance
Insufficient space on the root filesystem is more than a capacity alert. When the system cannot write temporary files, logs, or package data, services may stop, updates may fail, and both databases and monitoring systems may stop recording information. The correct approach is to confirm the impact and preserve evidence first, then gradually reclaim space that can be recovered safely. Do not simply delete a file because it is large.
Applicable Scenarios and Incident Classification
Use this runbook when an alert reports high root filesystem utilization, low remaining space, or write failures. Escalate the incident and suspend nonessential deployments if a critical service has stopped, the filesystem has become read-only, or free space continues to fall rapidly. If utilization is only approaching a warning threshold and the trend is stable, work can be scheduled for a maintenance window. Even then, the source of growth must be identified; the incident should not be closed after cleanup alone.
Step 1: Confirm the Scope
Use filesystem capacity tools to identify the affected mount point and check whether inodes are exhausted. A large number of small files can prevent new files from being created even when capacity remains available. Confirm whether the root filesystem and data disks are separate to avoid a false diagnosis. Record the incident time, utilization, service status, and recent changes, and retain screenshots or command output from before remediation for subsequent analysis.
Step 2: Identify the Source of Utilization
Start one level below the root directory and use capacity statistics restricted to the same filesystem. Narrow the search level by level until you identify package caches, system logs, application logs, temporary files, container data, or user files. Avoid scanning every network mount and virtual filesystem at the outset. If directory totals differ significantly from filesystem usage, check for files that have been deleted but remain open by running processes.
Do not look only for the largest files. Use modification times to determine whether a file is growing rapidly. Correlate the findings with log rotation, scheduled jobs, backups, and recent deployment records to determine why the data is growing, not merely what occupies the most space. If an application is continuously writing large volumes of data, reduce nonessential logging or suspend the relevant workload to prevent newly reclaimed space from being consumed immediately.
Step 3: Contain the Incident in a Safe Order
1. Clear the Package Cache
Downloaded files retained by the package manager can usually be retrieved again. Review their space usage first, then use the package manager's supported cleanup function. Before removing dependencies that are no longer considered necessary, review the list to ensure that no components required by business services will be deleted. Kernel packages should also be managed through the package-management process; do not delete system files directly.
2. Reduce System Log Usage
Review total log usage and the retention policy before reducing logs by age or size. Retention must meet incident-investigation and regulatory requirements; do not erase all evidence merely to free space immediately. If a single service is generating excessive logs, correct its log level, rotation, and compression settings, and confirm that the service can reopen its log file after rotation.
3. Manage Application and Temporary Data
The service owner must confirm retention and backup requirements for application logs. Temporary directories may also contain active workloads, so evaluate files by age, owner, and open status. Container images, stopped containers, and build caches can be managed with the platform's supported cleanup commands, but confirm before deletion that they are not required for recovery or audit purposes.
4. Handle Deleted Files That Remain Open
If a process still holds a deleted file open, the name disappears but the space is not released. First identify the process and assess the service impact, then close the file descriptor through a controlled service restart. Do not terminate an unidentified process arbitrarily. Databases and transaction-processing services must follow their normal shutdown procedures, with failover arranged when necessary.
Validation and Recovery
After each action, recheck free space and service status. This avoids losing the ability to trace the effect of individual actions after performing several cleanup operations at once. Confirm that package management, log writing, monitoring agents, and critical business functions have returned to normal. Observe the growth rate for a period of time and confirm that free space is not declining rapidly again before resolving the incident.
Long-Term Improvements
Establish multiple alert thresholds that cover absolute free capacity, inode availability, and growth rate in addition to utilization percentage. Configure log rotation, compression, size limits, and centralized retention. Separate application data from system disks where appropriate. Capacity forecasts should account for deployments, data growth, and retention policies. If demand is growing legitimately, expand the filesystem or reallocate storage instead of relying on repeated manual cleanup.
Key Risks
Accidentally deleting system files, database files, or unbacked-up data can cause an incident more serious than the original capacity shortage. Clearing logs destroys investigative evidence, and extensive scanning may increase disk load. Percentage-based thresholds alone can obscure the difference between a large disk with ample free space and a small partition approaching exhaustion. Thresholds should therefore combine absolute capacity with the rate of growth.
Checklist
- The affected mount point and inode status have been confirmed
- The impact, growth rate, and recent changes have been recorded
- The source has been isolated level by level without crossing into unrelated filesystems
- Owner, purpose, open status, and retention requirements are confirmed before deletion
- Supported cleanup mechanisms are used for packages, logs, and container data
- Free space and service health are validated after every cleanup step
- The root cause in rotation, retention, or abnormal write behavior has been corrected
- Monitoring covers capacity, inodes, and growth rate
Conclusion
The quality of a disk-space incident response is not measured by how quickly data is deleted, but by whether write capability can be restored with minimal risk while preserving traceable evidence. Following the sequence of confirming scope, locating the source, safely containing the incident, validating each step, and implementing long-term governance prevents repeated alerts and secondary incidents and turns a one-time emergency response into a capacity-management discipline.