No, after the reboot I'm still in the same situation. As mentioned earlier
I deleted /bsd.sp so I have a little more free space.
grits# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/sd0a 986M 916M 20.1M 98% /
/dev/sd0k 57.7G 25.9G 29.0G 47% /home
/dev/sd0d 3.9G 282K 3.7G 0% /tmp
/dev/sd0f 5.8G 1.1G 4.4G 21% /usr
/dev/sd0g 986M 234M 702M 25% /usr/X11R6
/dev/sd0h 16.8G 35.5M 15.9G 0% /usr/local
/dev/sd0j 5.8G 2.0K 5.5G 0% /usr/obj
/dev/sd0i 1.9G 2.0K 1.8G 0% /usr/src
/dev/sd0e 13.8G 19.0M 13.1G 0% /var
/dev/sd1c 440G 306G 112G 73% /backup
grits# du -xsh /
166M /
I take it I'm dealing with filesystem corruption as Ali mentioned earlier?
On Tue, Aug 3, 2021 at 11:10 PM Otto Moerbeek <otto@drijf.net> wrote:
> On Tue, Aug 03, 2021 at 10:57:42PM -0700, Greg Thomas wrote:
>
> > I thought Paul's advice only applies if I was trying to figure it out
> > before rebooting? I'd already rebooted before sending my first email.
>
> OK, did the free space come back in df after reboot? If so, then it's
> programs having open files that are unlinked for sure.
>
> -Otto
>
> >
> >
> >
> > On Tue, Aug 3, 2021 at 10:40 PM Otto Moerbeek <otto@drijf.net> wrote:
> >
> > > On Tue, Aug 03, 2021 at 12:39:54PM -0700, Greg Thomas wrote:
> > >
> > > > I'm definitely suffering from filesystem corruption on root. I had
> > > > rebooted last night with no change.
> > > >
> > > > I have no options for mounting root.
> > > >
> > > > grits# cat /etc/fstab
> > > > 16a27b4b4549ce04.b none swap sw
> > > > 16a27b4b4549ce04.a / ffs rw 1 1
> > > > 16a27b4b4549ce04.k /home ffs rw,nodev,nosuid 1 2
> > > > 16a27b4b4549ce04.d /tmp ffs rw,nodev,nosuid 1 2
> > > > 16a27b4b4549ce04.f /usr ffs rw,nodev 1 2
> > > > 16a27b4b4549ce04.g /usr/X11R6 ffs rw,nodev 1 2
> > > > 16a27b4b4549ce04.h /usr/local ffs rw,wxallowed,nodev 1 2
> > > > 16a27b4b4549ce04.j /usr/obj ffs rw,nodev,nosuid 1 2
> > > > 16a27b4b4549ce04.i /usr/src ffs rw,nodev,nosuid 1 2
> > > > 16a27b4b4549ce04.e /var ffs rw,nodev,nosuid 1 2
> > > > /dev/sd1c /backup ffs rw,nodev,nosuid 1 2
> > > >
> > > > I need to upgrade so I can do that from scratch. This is my backup
> > > server
> > > > so the configuration is pretty simple.
> > > >
> > > > Not sure fsck output helps here?
> > > >
> > > > grits# fsck /dev/sd0a
> > > > ** /dev/rsd0a (NO WRITE)
> > > > ** Last Mounted on /
> > > > ** Root file system
> > > > ** Phase 1 - Check Blocks and Sizes
> > > > ** Phase 2 - Check Pathnames
> > > > ** Phase 3 - Check Connectivity
> > > > ** Phase 4 - Check Reference Counts
> > > > ** Phase 5 - Check Cyl groups
> > > > 12852 files, 469195 used, 35516 free (44 frags, 4434 blocks, 0.0%
> > > > fragmentation)
> > > >
> > > > Anyway, I'll reinstall unless someone has more learning experiences
> for
> > > me.
> > > >
> > > > And thank you to Paul for giving a quick explanation of the
> difference
> > > > between df and du.
> > > >
> > > > Thanks all!
> > >
> > > fsck looks normal for a mounted filesystem.
> > >
> > > but did you try following Paul's advice to find an open file that has
> > > no directory entry? That is not corruption, but explains why more
> > > storage is in use than du shows.
> > >
> > > -Otto
> > >
> > > >
> > > >
> > > >
> > > > On Tue, Aug 3, 2021 at 11:39 AM Ali Farzanrad <
> ali_farzanrad@riseup.net>
> > > > wrote:
> > > >
> > > > > I also suspected that it is a filesystem corruption.
> > > > > Do you have `async` mount option on your root?
> > > > >
> > > > > Sebastien Marie <semarie@online.fr> wrote:
> > > > > > On Tue, Aug 03, 2021 at 10:03:44AM +0200, Paul de Weerd wrote:
> > > > > > > df shows you how much data you can write to an fs, while du
> shows
> > > the
> > > > > > > disk usage of files it can find. If it can't find a file
> (because
> > > > > > > it's been deleted), it won't account for it. But if it's been
> > > deleted
> > > > > > > and still held open by some process, it would still consume
> disk
> > > > > > > space.
> > > > > > >
> > > > > > > So it looks like a process has a file open on the root
> filesystem
> > > that
> > > > > > > has been deleted. You're looking for a root-owned process
> that is
> > > > > > > (probably) long-running. My guess the file is in /dev/
> (that's my
> > > > > > > crystal ball talking though).
> > > > > > >
> > > > > > > Easiest way out is generally to reboot - this stops all
> processes
> > > > > > > (d0h), dus freeing up all the resources they had tied up,
> including
> > > > > > > files that had been deleted from the filesystem. But going
> through
> > > > > > > your process list to see if you can spot something that may
> have
> > > done
> > > > > > > this can be a good learning experience. In general, base
> OpenBSD
> > > > > > > daemons don't behave this way.
> > > > > >
> > > > > > I agree with Paul: you should have a running process which hold
> > > > > > descriptor on unlinked file.
> > > > > >
> > > > > > fstat(1) could be used to see list of opened files, and specially
> > > > > > unlinked files:
> > > > > >
> > > > > > INUM The inode number of the file. It will be followed
> by an
> > > > > asterisk
> > > > > > ('*') if the inode is unlinked from disk.
> > > > > >
> > > > > >
> > > > > > $ fstat | grep -F '* -'
> > > > > > [...]
> > > > > > semarie chrome 537 25 /tmp 48* -rw-------
> rwp
> > > > > 279793
> > > > > > [...]
> > > > > >
> > > > > > here, chrome (pid 537) has descriptor 25 opened to a file on /tmp
> > > > > > inode=48 (unlinked), the file size is 279793 bytes.
> > > > > >
> > > > > > --
> > > > > > Sebastien Marie
> > > > > >
> > > > > >
> > > > >
> > > > >
> > >
>
No comments:
Post a Comment