Today I had a problem with a server that had no more disk space. And I learned something new in the process.

df -h told me that 100% was in use. Which was about 29GB on that server.

But if I checked the root partition with du -shx / i got about 9GB of used space.

So off to checking where the space could have gone:

Inode usage

I know “du” does not take account for inodes, etc. But according to dumpe2fs /dev/sdx1 my Inode size * Inode count = about 700MB.
So that was not it.

“Hidden” directories under mountpoints

“du” will not see used space of files located in a path that is later mounted to another file-system. For example, if you have files in /home/ on your root partition, but has later mounted /home to its own partition. The files will be hidden behind the new mountpoint.

To be able to check these files without unmounting anything in use, I did the following:

mount --bind / /mnt
du -shx /mnt


If “du” would give me a different result now, I would have known that the files where hidden under one of my mountpoints. But sadly, they where not. I was starting to run out of options.

Deleted files

If a process opens a file, and then you delete it by rm thefile you will have the file deleted from the filesystem, but the inodes will not be freed before the process closes/releases the file. This is something I love about Linux/Posix systems, since that means that processes does not need to lock my files and I have full control over my files as opposed to other operating systems(windows). But I thought that when you delete a opened file, there is no easy way of knowing which deleted files are “held back” by processes. But there is!

lsof | grep deleted quickly gave me a list of files that has been deleted, but is held open by processes, and their size. In my case a deleted log file of 17GB in size was held by asterisk. So i reloaded the logger module of asterisk, and hey presto! The diskspace was available again.

Now only 9GB was “in use” according to df -h.