{"id":255618,"date":"2016-10-08T15:00:15","date_gmt":"2016-10-08T07:00:15","guid":{"rendered":"http:\/\/blog.zhenglei.net\/?p=255618"},"modified":"2016-10-08T15:00:15","modified_gmt":"2016-10-08T07:00:15","slug":"missing-diskspace-in-linux","status":"publish","type":"post","link":"https:\/\/blog.zhenglei.net\/?p=255618","title":{"rendered":"Missing diskspace in Linux"},"content":{"rendered":"

http:\/\/lunatic.no\/2014\/08\/missing-diskspace-in-linux\/<\/a><\/p>\n

Missing diskspace in Linux<\/h1>\n
\n
\n
<\/div>\n<\/div>\n

Today I had a problem with a server that had no more disk space. And I learned something new in the process.<\/p>\n

df -h<\/code> told me that 100% was in use. Which was about 29GB on that server.<\/p>\n

But if I checked the root partition with du -shx \/<\/code> i got about 9GB of used space.<\/p>\n

So off to checking where the space could have gone:<\/p>\n

Inode usage<\/h2>\n

I know \u201cdu\u201d does not take account for inodes, etc. But according to dumpe2fs \/dev\/sdx1<\/code> my Inode size * Inode count = about 700MB.
\nSo that was not it.<\/p>\n

\u201cHidden\u201d directories under mountpoints<\/h2>\n

\u201cdu\u201d will not see used space of files located in a path that is later mounted to another file-system. For example, if you have files in \/home\/ on your root partition, but has later mounted \/home to its own partition. The files will be hidden behind the new mountpoint.<\/p>\n

To be able to check these files without unmounting anything in use, I did the following:
\n
\nmount --bind \/ \/mnt
\ndu -shx \/mnt
\n<\/code><\/p>\n

If \u201cdu\u201d would give me a different result now, I would have known that the files where hidden under one of my mountpoints. But sadly, they where not. I was starting to run out of options.<\/p>\n

Deleted files<\/h2>\n

If a process opens a file, and then you delete it by rm thefile<\/code> you will have the file deleted from the filesystem, but the inodes will not be freed before the process closes\/releases the file. This is something I love about Linux\/Posix systems, since that means that processes does not need to lock my files and I have full control over my files as opposed to other operating systems(windows)<\/small>. But I thought that when you delete a opened file, there is no easy way of knowing which deleted files are \u201cheld back\u201d by processes. But there is!<\/p>\n

lsof | grep deleted<\/code> quickly gave me a list of files that has been deleted, but is held open by processes, and their size. In my case a deleted log file of 17GB in size was held by asterisk. So i reloaded the logger module of asterisk, and hey presto! The diskspace was available again.<\/p>\n

Now only 9GB was \u201cin use\u201d according to df -h.<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

http:\/\/lunatic.no\/2014\/08\/missing-disksp … \u7ee7\u7eed\u9605\u8bfb →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[320],"class_list":["post-255618","post","type-post","status-publish","format-standard","hentry","category-linux","tag-disk"],"_links":{"self":[{"href":"https:\/\/blog.zhenglei.net\/index.php?rest_route=\/wp\/v2\/posts\/255618","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.zhenglei.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.zhenglei.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.zhenglei.net\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.zhenglei.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=255618"}],"version-history":[{"count":1,"href":"https:\/\/blog.zhenglei.net\/index.php?rest_route=\/wp\/v2\/posts\/255618\/revisions"}],"predecessor-version":[{"id":255619,"href":"https:\/\/blog.zhenglei.net\/index.php?rest_route=\/wp\/v2\/posts\/255618\/revisions\/255619"}],"wp:attachment":[{"href":"https:\/\/blog.zhenglei.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=255618"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.zhenglei.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=255618"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.zhenglei.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=255618"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}