/ home directory full again

Hi all,

I am working on a Sun OS v5.7. I have a home directory that is saying that it's full, even though there are only a few actual files within the different user directories. The o/p is below after using df -bk.

....
/dev/dsk/c0t0d0s3 192833 108410 65140 63% /var
/dev/dsk/c0t0d0s7 4002513 161572 3800916 5% /free
/dev/dsk/c0t0d0s4 482824 434542 0 100% /home
/dev/dsk/c0t0d0s5 482824 268810 165732 62% /opt
.....


Daemonguy, the moderator had suggested the following:


I have to mention of course, that I have seen hacked boxes which use various filesystems to store live traffic data prior to sending it out to an IRC channel. You may want to watch the connections with some permutation of netstat, or even install something like Snort to try and find a clue. Of course, crafting a tcpdump script should also suffice.


Could someone tell me how would I go about finding this out ? This is happened for the 2nd time now & it seems that the above suggested scenario is likely ...

V

What have you done to look at the files? Try du -h to see a listing of all the files in the directory, and how much space they take up. That should help, for a start.

Hey all,

It seems that there are no unecessary files or data within the home directory. Bu the directory shows 100% full all the time.

Restarting the server has solved the issue on both the occasions that it has happened. Could it be a memory allocation problem ? I ran 'vmstat' at that time and could see nothing worng there either ...

V

Hmm, well often garbage collection is done at reboot, but that would mean that someplace in /home contains rapidly collecting garbage files. (Everything from java collections to temp files).

It could also be related to updates. Common on web servers, but anything that writes large amounts or large files can cause it.
Here's how it works;
If something is writing large files (or large numbers of files) and something else (or the program itself) is using those files, a filesystem will delete the files as requested -- thus showing up in df as gone -- however the program has not released the file until IT is finished with it(them).
This could be why you see it fixed when you reboot, system does clean-up at reboot.
Starting and stopping the program which is making use of said files would help, but then you are not sure what that might be. Merely deleting files which are in use does not alleviate the issue.
Putting a syslog rotator in place helps the situation, but first you have to find out what -- if anything -- is writing to the home dir, or some recursive dir contained therein.

The first thing I would do is examine every running userland proc, and find out form their configs where files -- if any -- get written. If any start out with /home OR if they write to someplace else which is sym-linked to someplace in the home dir space, you may have found your culprit.

That's my best guess at this point. Without doing any more intensive investigation anyway.

Cheers.