Disk Space: Filling Up, and Where It Actually Went
System Monitoring & Performance Diagnosis
Chapter 5 · Disk Space: Filling Up, and Where It Actually Went
Chapter 4 covered disk performance — how fast a disk can service requests. This chapter covers a different, equally common problem: a disk running out of space entirely. The two headline tools, df and du, usually agree — until they don't, and the gap between them turns out to be this chapter's own most useful diagnostic finding.
df: The Quick Overview
df reports space usage at the filesystem level — fast, and a good first check, but it says nothing about which files or directories are actually responsible.
du: Finding Where the Space Actually Went
du walks the actual directory tree, adding up real file sizes, and is the natural next step once df has confirmed a filesystem is genuinely full.
The Classic df/du Mismatch
Adding up the du figures above comes to roughly 30 GB. df reported 49 GB used on the same filesystem. That's not a rounding error — it's a real, specific, and genuinely common phenomenon.
du walks the visible directory tree by name, and a deleted file has no name left to walk to, so it's invisible to du entirely. df, which reports actual block-level usage, still counts it. The gap between the two totals is real, allocated disk space, held open by a process, with no visible trace in the filesystem tree at all.
Finding the Phantom Space
lsof +L1 lists open files with a link count of 0 — files that have been deleted but are still held open by a running process, exactly this chapter's own missing gap. The SIZE/OFF column here, roughly 18 GB, accounts for almost the entire shortfall between df and du's totals.
log1) course covers the proper, proactive fix — logrotate, which handles this correctly — in its own Chapter 9. This chapter is about diagnosing the symptom once it's already happened, not preventing it in the first place.
Windows' Own Disk-Usage Story
Windows' built-in tooling for "where did my space actually go" is comparatively weak — Storage settings gives only a rough, high-level breakdown by category, not a real directory-by-directory view. Third-party tools like WinDirStat fill that gap with a genuine visual breakdown, similar in spirit to du. A related situation to the deleted-but-open-file gotcha can occur on Windows too — a file with an open handle can resist being fully reclaimed even after deletion — though NTFS's own file-locking model differs enough from Linux's link-count semantics that the two aren't a precise match.
When Disk Fills Completely
A filesystem that reaches 100% doesn't just stop growing gracefully — a database can refuse further writes, a web server can fail to log or even serve requests, and in severe cases the operating system itself can struggle, since some system services genuinely need a small amount of free space to operate at all. This is worth treating with real urgency once a filesystem is closing in on full, not just noted and revisited later.
Working Example: The Missing 18 GB
A ticket: /var is at 98% and climbing, but a directory walk only accounts for about 30 GB of the 49 GB df reports as used. lsof +L1 finds the answer directly: a Java application still holding open a deleted 18 GB log file — deleted by an overnight cleanup script that never told the running process to reopen a fresh one. Restarting the application (or, less disruptively, sending it a signal to reopen its log handles, if it supports one) releases the space immediately, confirmed by a fresh df -h showing usage drop back in line with what du was already reporting.
Hands-On Exercises
Explain why du's totals can add up to noticeably less than what df reports as used on the same filesystem, using this chapter's own explanation.
Explain what lsof +L1 specifically looks for, and why that's exactly the right tool for the df/du mismatch this chapter describes.
Explain what caused the missing 18 GB in this chapter's worked example, and why simply deleting a large log file directly (instead of using something like logrotate) can lead to exactly this situation.
📄 View solutionChapter 5 Quick Reference
df -hshows overall filesystem usage;du -shshows where it lives in the directory tree- A deleted file still held open by a running process occupies real space invisible to
dubut still counted bydf— the classic mismatch lsof +L1finds exactly these deleted-but-open files directly- The proactive fix (
logrotate) is covered in Logging & Log Analysis (log1) Chapter 9 — this chapter diagnoses the symptom after the fact - Windows' built-in disk-usage tooling is comparatively weak — third-party tools like WinDirStat fill the gap
- A completely full disk has real downstream consequences — database write failures, logging failures, even OS-level instability
- Next chapter: Network as a Performance Symptom