Space is not freed after removing the file in Unix

Written by Kokei's Blog
Posted May 19, 2011 at 12:34 AM

This situation usually occurs when a file is deleted while it is still opened by some other processes. This is because when open a file opened by a process, the reference count of the file will increment by 1. Removing the file only remove the directory point to that file and it does not remove the underlying file opened by the process.

Only until the offending processes close the file, either explicitly close the file with close() or manually kill the process, the reference to the file returns to zero and space occupied by the file is freed. A common example is Oracle writing to a trace files and these trace files grows to be large. When a DBA try to free the space by removing the trace files, he may found df command does not reflect the correct result. Under most circumstances, the most easiest way to free the space from the trace files is bounce the Oracle instance.

Below is an example performed 0n Solaris 10 to demonstrate how the space allocated to a file is not freed after removing the file.

First we check the disk usage of the tmp diretory.

testsvr$ df -h|grep -i /tmp

swap 78G 488K 78G 1% /tmp

We run dd command to create a file named testfile in /tmp in the background. Since we are creating a 30GB file, it takes some time complete and the background process will keep the file open.

testsvr$ nohup dd if=/dev/zero of=/tmp/testfile bs=1048576 count=307200 &

[3] 17460

testsvr$ Sending output to nohup.out

testsvr$ ls -al /tmp/testfile

-rw-r--r-- 1 oracle dba 1144528896 May 18 23:42 /tmp/testfile

testsvr$ df -h|grep -i /tmp

swap 78G 1.9G 77G 3% /tmp

Here we can see that the dd command is still running and the size of the /tmp/testfile is growing

testsvr$ ps -ef |grep dd

oracle 17841 12384 0 23:42:15 pts/7 0:00 grep dd

oracle 17460 12384 2 23:42:00 pts/7 0:15 dd if=/dev/zero of=/tmp/testfile bs=1048576 count=307200

Then we delete the file.

testsvr$ rm /tmp/testfile

testsvr$

testsvr$ ls -al /tmp/testfile

/tmp/testfile: No such file or directory

We spotted the process id 17460 is opening a file with no named filename linked to it.

testsvr$ find /proc/*/fd -type f -links 0 \! -size 0 -ls

67 8 --w------- 0 oracle dba 639 Dec 9 05:16 /proc/1328/fd/5

427559811 6944224 --w------- 0 oracle dba 7110885376 May 18 23:42 /proc/17460/fd/4

67 8 --w------- 0 oracle dba 639 Dec 9 05:16 /proc/25269/fd/5

31 8 --w------- 0 oracle dba 1418 Apr 16 11:03 /proc/25280/fd/2

67 8 --w------- 0 oracle dba 639 Dec 9 05:16 /proc/25280/fd/5

67 8 --w------- 0 oracle dba 639 Dec 9 05:16 /proc/25286/fd/5

67 8 --w------- 0 oracle dba 639 Dec 9 05:16 /proc/25297/fd/5

67 8 --w------- 0 oracle dba 639 Dec 9 05:16 /proc/25310/fd/5

67 8 --w------- 0 oracle dba 639 Dec 9 05:16 /proc/25322/fd/5

67 8 --w------- 0 oracle dba 639 Dec 9 05:16 /proc/25326/fd/5

102 8 --w------- 0 oracle dba 5015 May 13 14:25 /proc/25332/fd/2

testsvr$ df -h|grep -i /tmp

swap 78G 10G 68G 14% /tmp

Then we kill the process 17460.

testsvr$ kill -9 17460