softdep / softraid RAID1 issue?

Hiya.

tl;dr version: multiple machines with softraid RAID1 & softdep have
file systems freeze when doing lots of I/O, possibly involving
adding and removing links from the same files at the same time.
Workaround found.

Need help finding better diagnostic information.


long version:
=============
I have a couple systems which have had an issue for a long time
where suddenly, disk activity would just ... stop. No message on
console, no panic. Usually, I can still log in, but if I touch
the effected file systems, my SSH session (or console login) freezes.
Always happens during some intense disk activity (more on that in a
moment). When it happens, I can not reboot the system without a hard
reset or power cycle (and these systems have multi-TB file systems on
them, so doing this is painful). umount on the impacted file systems
hangs.

I'm not really sure if I lose all file systems or just some. Most of
the file systems are very "static". Some things seem to work for a
while, but it could well be just cached data.

I tried swapping computers, replacing disks, and even doing weekly
reboots, all to seemingly no impact. Problem has occurred for well
over a year, maybe longer. Upgraded frequently to most recent
snaphot, no change seen (I often use hangs as an opporunity to
upgrade).

Recently, however, I think I caught it in mid-failure. Disk activity
was still going on, but it was very slow. 'top' showed "WAIT" on
softdep (or something similar). The jobs that should have been long
done was still running according to the logs (new files being added
to the list of files backed up), but very very slowly. And then it
came to a hard stop, as before. This may have been an unusual event,
or I might just have happened to look in right place to see something
was still happening.

These machines are used for rsync --link-dest backup. The short
version of the algorithm is something like this:

=-------------------------
PREVIOUS=(find previous backup)
TODAY=(today's date)
OLDEST=(find oldest backup in the set)
REMOTE=(machine we are backing up)

# remove oldest backup
rm -r $OLDEST &

mkdir $TODAY

# make new backup
rsync --link-dest $PREVIOUS $REMOTE $TODAY
=-------------------------

This backup process basically makes a hard link for files that haven't
changed, and copies over files that did change. After first backup,
all future backups are incrementals, both in time and additional disk
usage.

A bunch of these will typically be running at the same time, maybe five
to ten of them (adjusting this number didn't seem to have any effect).
When it fails, usually a few succeed, and the rest just never complete.

Here's where it gets weird -- removing the '&' after the rm -r $OLDEST
line seems to have FIXED THE PROBLEM. No problems in 18 days, which is
a pretty good record.

SPECULATION:
the rm and rsync processes running at the same time can potentially be
putting both new links and removing old links from the same file at the
same time (well...multi-tasking definition of "same time"). Maybe
something is having a problem with this.

I have another machine running the same backup process, which has not
had a problem. It has been running happily for 123 days now (yeah, I
kinda forgot about it). It is a little laptop, so only one hard disk,
but I am using a softraid encrypted disk. So yes, also using softraid,
but only one media to read/write to. So maybe associated with either
RAID1 or multiple disk I/Os.

So, my problem is worked around, but I suspect there's still a bug
there.

I am happy to put the '&' back and gather more information next time
it happens...if someone tells me what info to gather.

Nick.


Machine that has had problems, but fixed by no longer backgrounding
the rm -r $OLDEST backup:

OpenBSD 7.3-current (GENERIC.MP) #1175: Wed May 3 08:19:33 MDT 2023
deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 5872685056 (5600MB)
avail mem = 5675061248 (5412MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xeb240 (42 entries)
bios0: vendor AMI version "7.16" date 01/18/2012
bios0: Hewlett-Packard p6-2108p
acpi0 at bios0: ACPI 4.0
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP APIC MCFG SLIC HPET SSDT SSDT DBGP
acpi0: wakeup devices SBAZ(S4) P0PC(S4) UHC1(S3) UHC2(S3) USB3(S3) UHC4(S3) USB5(S3) UHC6(S3) UHC7(S3) XHC0(S3) XHC1(S3) PE20(S4) PE21(S4) PE22(S4) PE23(S4) BR15(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD E2-3200 APU with Radeon(tm) HD Graphics, 2395.89 MHz, 12-01-00
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,3DNOW2,3DNOW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,ITSC
cpu0: 64KB 64b/line 2-way D-cache, 64KB 64b/line 2-way I-cache
cpu0: 512KB 64b/line 16-way L2 cache
cpu0: smt 0, core 0, package 0
cpu0: AMD erratum 721 detected and fixed
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 199MHz
cpu0: mwait min=64, max=64, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: AMD E2-3200 APU with Radeon(tm) HD Graphics, 2396.04 MHz, 12-01-00
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,3DNOW2,3DNOW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,ITSC
cpu1: 64KB 64b/line 2-way D-cache, 64KB 64b/line 2-way I-cache
cpu1: 512KB 64b/line 16-way L2 cache
cpu1: smt 0, core 1, package 0
ioapic0 at mainbus0: apid 3 pa 0xfec00000, version 21, 24 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0xe0000000, bus 0-255
acpihpet0 at acpi0: 14318180 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (PE20)
acpiprt2 at acpi0: bus -1 (PE21)
acpiprt3 at acpi0: bus -1 (PE22)
acpiprt4 at acpi0: bus -1 (PE23)
acpiprt5 at acpi0: bus 1 (BR12)
acpiprt6 at acpi0: bus 2 (BR14)
acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
acpicmos0 at acpi0
acpibtn0 at acpi0: PWRB
acpicpu0 at acpi0: C2(0@100 io@0x1771), C1(@1 halt!), PSS
acpicpu1 at acpi0: C2(0@100 io@0x1771), C1(@1 halt!), PSS
cpu0: 2395 MHz: speeds: 2400 2100 1900 1600 1400 1200 1000 800 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "AMD 12h Host" rev 0x00
radeondrm0 at pci0 dev 1 function 0 "ATI Radeon HD 6370D" rev 0x00
drm0 at radeondrm0
radeondrm0: msi
ppb0 at pci0 dev 2 function 0 "AMD 12h PCIE" rev 0x00: msi
pci1 at ppb0 bus 1
ppb1 at pci0 dev 4 function 0 "AMD 12h PCIE" rev 0x00: msi
pci2 at ppb1 bus 2
ppb2 at pci0 dev 5 function 0 "AMD 12h PCIE" rev 0x00: msi
pci3 at ppb2 bus 3
re0 at pci3 dev 0 function 0 "Realtek 8168" rev 0x06: RTL8168E/8111E (0x2c00), msi, address e8:40:f2:10:18:3d
rgephy0 at re0 phy 7: RTL8169S/8110S/8211 PHY, rev. 4
ahci0 at pci0 dev 17 function 0 "AMD Hudson-2 SATA" rev 0x40: msi, AHCI 1.3
ahci0: port 0: 3.0Gb/s
ahci0: port 1: 1.5Gb/s
ahci0: port 2: 3.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, HGST HUS724040AL, MFAO> naa.5000cca249d26e65
sd0: 3815447MB, 512 bytes/sector, 7814037168 sectors
cd0 at scsibus1 targ 1 lun 0: <hp, DVD A DH16ABSH, YHDD> removable
sd1 at scsibus1 targ 2 lun 0: <ATA, HGST HUS724040AL, MFAO> naa.5000cca23de718a6
sd1: 3815447MB, 512 bytes/sector, 7814037168 sectors
ohci0 at pci0 dev 18 function 0 "AMD Hudson-2 USB" rev 0x11: apic 3 int 18, version 1.0, legacy support
ehci0 at pci0 dev 18 function 2 "AMD Hudson-2 USB2" rev 0x11: apic 3 int 17
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "AMD EHCI root hub" rev 2.00/1.00 addr 1
ohci1 at pci0 dev 19 function 0 "AMD Hudson-2 USB" rev 0x11: apic 3 int 18, version 1.0, legacy support
ehci1 at pci0 dev 19 function 2 "AMD Hudson-2 USB2" rev 0x11: apic 3 int 17
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "AMD EHCI root hub" rev 2.00/1.00 addr 1
piixpm0 at pci0 dev 20 function 0 "AMD Hudson-2 SMBus" rev 0x13: SMI
iic0 at piixpm0
spdmem0 at iic0 addr 0x52: 2GB DDR3 SDRAM PC3-10600
spdmem1 at iic0 addr 0x53: 4GB DDR3 SDRAM PC3-10600
azalia0 at pci0 dev 20 function 2 "AMD Hudson-2 HD Audio" rev 0x01: apic 3 int 16
azalia0: codecs: Realtek ALC662
audio0 at azalia0
pcib0 at pci0 dev 20 function 3 "AMD Hudson-2 LPC" rev 0x11
ppb3 at pci0 dev 20 function 4 "AMD Hudson-2 PCI" rev 0x40
pci4 at ppb3 bus 4
ohci2 at pci0 dev 20 function 5 "AMD Hudson-2 USB" rev 0x11: apic 3 int 18, version 1.0, legacy support
sdhc0 at pci0 dev 20 function 7 "AMD Hudson-2 SD Host Controller" rev 0x00: apic 3 int 16
sdhc0: SDHC 2.0, 50 MHz base clock
sdmmc0 at sdhc0: 4-bit, sd high-speed, mmc high-speed, dma
ohci3 at pci0 dev 22 function 0 "AMD Hudson-2 USB" rev 0x11: apic 3 int 18, version 1.0, legacy support
ehci2 at pci0 dev 22 function 2 "AMD Hudson-2 USB2" rev 0x11: apic 3 int 17
usb2 at ehci2: USB revision 2.0
uhub2 at usb2 configuration 1 interface 0 "AMD EHCI root hub" rev 2.00/1.00 addr 1
pchb1 at pci0 dev 24 function 0 "AMD 14h Link Cfg" rev 0x43
pchb2 at pci0 dev 24 function 1 "AMD 14h Address Map" rev 0x00
pchb3 at pci0 dev 24 function 2 "AMD 14h DRAM Cfg" rev 0x00
km0 at pci0 dev 24 function 3 "AMD 14h Misc Cfg" rev 0x00
pchb4 at pci0 dev 24 function 4 "AMD 14h CPU Power" rev 0x00
pchb5 at pci0 dev 24 function 5 "AMD 14h Reserved" rev 0x00
pchb6 at pci0 dev 24 function 6 "AMD 14h NB Power" rev 0x00
pchb7 at pci0 dev 24 function 7 "AMD 14h Reserved" rev 0x00
usb3 at ohci0: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "AMD OHCI root hub" rev 1.00/1.00 addr 1
usb4 at ohci1: USB revision 1.0
uhub4 at usb4 configuration 1 interface 0 "AMD OHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
usb5 at ohci2: USB revision 1.0
uhub5 at usb5 configuration 1 interface 0 "AMD OHCI root hub" rev 1.00/1.00 addr 1
usb6 at ohci3: USB revision 1.0
uhub6 at usb6 configuration 1 interface 0 "AMD OHCI root hub" rev 1.00/1.00 addr 1
vmm0 at mainbus0: SVM/RVI
umass0 at uhub1 port 1 configuration 1 interface 0 "Generic Mass Storage Device" rev 2.00/1.00 addr 2
umass0: using SCSI over Bulk-Only
scsibus2 at umass0: 2 targets, initiator 0
sd2 at scsibus2 targ 1 lun 0: <Multiple, Card Reader, 1.00> removable serial.058f6366058F63666433
uhidev0 at uhub3 port 3 configuration 1 interface 0 "Chicony USB Keyboard" rev 1.10/3.90 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd0 at ukbd0: console keyboard
uhidev1 at uhub3 port 3 configuration 1 interface 1 "Chicony USB Keyboard" rev 1.10/3.90 addr 2
uhidev1: iclass 3/0, 2 report ids
ucc0 at uhidev1 reportid 1: 8 usages, 1 key, enum
wskbd1 at ucc0 mux 1
uhid0 at uhidev1 reportid 2: input=1, output=0, feature=4
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
sd3 at scsibus4 targ 1 lun 0: <OPENBSD, SR RAID 1, 006>
sd3: 3815447MB, 512 bytes/sector, 7814036576 sectors
root on sd3a (5515d13b42e30c29.a) swap on sd3b dump on sd3b
radeondrm0: SUMO2
radeondrm0: 1024x768, 32bpp
wsdisplay0 at radeondrm0 mux 1: console (std, vt100 emulation), using wskbd0
wskbd1: connecting to wsdisplay0
wsdisplay0: screen 1-5 added (std, vt100 emulation)

$ mount
/dev/sd3a on / type ffs (local, noatime, softdep)
/dev/sd3h on /home type ffs (local, noatime, nodev, nosuid, softdep)
/dev/sd3f on /tmp type ffs (local, noatime, nodev, nosuid, softdep)
/dev/sd3d on /usr type ffs (local, noatime, nodev, softdep)
/dev/sd3e on /usr/local type ffs (local, noatime, nodev, wxallowed, softdep)
/dev/sd3g on /var type ffs (local, noatime, nodev, nosuid, softdep)
/dev/sd3p on /bu type ffs (local, noatime, nodev, nosuid, softdep)
/dev/sd3i on /v/1 type ffs (local, noatime, nodev, noexec, nosuid, softdep)
/dev/sd3j on /v/2 type ffs (local, noatime, nodev, noexec, nosuid, softdep)

=-------------------------

Machine that hasn't had an issue:
OpenBSD 7.2-current (GENERIC.MP) #780: Wed Oct 12 21:11:59 MDT 2022
deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 3706445824 (3534MB)
avail mem = 3576709120 (3411MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe4800 (23 entries)
bios0: vendor Insyde version "F.14" date 12/06/2013
bios0: Hewlett-Packard HP 215 G1 Notebook PC
acpi0 at bios0: ACPI 5.0
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP UEFI HPET APIC MCFG ASF! BOOT SLIC WDRT WDAT FPDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT
acpi0: wakeup devices XPDV(S5) OHC1(S3) OHC2(S3) OHC3(S3) EHC1(S3) EHC2(S3) EHC3(S3) XHC0(S3) LID_(S4)
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpihpet0 at acpi0: 14318180 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD A4-1250 APU with Radeon(TM) HD Graphics, 998.14 MHz, 16-00-01
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 2-way I-cache, 1MB 64b/line 16-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: AMD A4-1250 APU with Radeon(TM) HD Graphics, 998.13 MHz, 16-00-01
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 2-way I-cache, 1MB 64b/line 16-way L2 cache
cpu1: smt 0, core 1, package 0
ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 21, 24 pins, remapped
ioapic1 at mainbus0: apid 5 pa 0xfec01000, version 21, 32 pins, remapped
acpimcfg0 at acpi0
acpimcfg0: addr 0xf8000000, bus 0-63
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (GPP0)
acpiprt2 at acpi0: bus 1 (GPP1)
acpiprt3 at acpi0: bus 2 (GPP2)
acpiprt4 at acpi0: bus 3 (GPP3)
acpiprt5 at acpi0: bus -1 (GFX_)
acpiec0 at acpi0
acpibtn0 at acpi0: PWRB
acpipci0 at acpi0 PCI0
acpicmos0 at acpi0
acpibat0 at acpi0: BAT1 model "KP03036" serial 04320 11/19/2013 type LION oem "11-83"
acpiac0 at acpi0: AC unit online
"HPQ6007" at acpi0 not configured
acpibtn1 at acpi0: LID_
"PNP0C14" at acpi0 not configured
"HPQ6001" at acpi0 not configured
"ASD0001" at acpi0 not configured
acpicpu0 at acpi0: C2(0@400 io@0x414), C1(@1 halt!), PSS
acpicpu1 at acpi0: C2(0@400 io@0x414), C1(@1 halt!), PSS
acpitz0 at acpi0: critical temperature is 226 degC
acpitz1 at acpi0: no critical temperature defined
acpivideo0 at acpi0: VGA_
acpivout0 at acpivideo0: LCD_
acpivideo1 at acpi0: VGA_
cpu0: 998 MHz: speeds: 1000 900 800 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "AMD 16h Host" rev 0x00
radeondrm0 at pci0 dev 1 function 0 "ATI Kabini" rev 0x00
drm0 at radeondrm0
radeondrm0: msi
azalia0 at pci0 dev 1 function 1 "ATI Radeon HD Audio" rev 0x00: msi
azalia0: no supported codecs
pchb1 at pci0 dev 2 function 0 vendor "AMD", unknown product 0x1538 rev 0x00
ppb0 at pci0 dev 2 function 3 "AMD 16h PCIE" rev 0x00: msi
pci1 at ppb0 bus 1
re0 at pci1 dev 0 function 0 "Realtek 8101E" rev 0x07: RTL8106E (0x4480), msi, address a0:1d:48:12:ab:6c
rlphy0 at re0 phy 7: RTL8201E 10/100 PHY, rev. 2
ppb1 at pci0 dev 2 function 4 "AMD 16h PCIE" rev 0x00: msi
pci2 at ppb1 bus 2
vendor "Broadcom", unknown product 0x4359 (class network subclass miscellaneous, rev 0x00) at pci2 dev 0 function 0 not configured
ppb2 at pci0 dev 2 function 5 "AMD 16h PCIE" rev 0x00: msi
pci3 at ppb2 bus 3
rtsx0 at pci3 dev 0 function 0 "Realtek RTS5229 Card Reader" rev 0x01: msi
sdmmc0 at rtsx0: 4-bit, dma
xhci0 at pci0 dev 16 function 0 "AMD Bolton xHCI" rev 0x01: msi, xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "AMD xHCI root hub" rev 3.00/1.00 addr 1
ahci0 at pci0 dev 17 function 0 "AMD Hudson-2 SATA" rev 0x00: msi, AHCI 1.3
ahci0: port 0: 3.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, TOSHIBA MQ01ABF0, AM00> naa.5000039545684868
sd0: 305245MB, 512 bytes/sector, 625142448 sectors
ohci0 at pci0 dev 18 function 0 "AMD Hudson-2 USB" rev 0x39: apic 4 int 18, version 1.0, legacy support
ehci0 at pci0 dev 18 function 2 "AMD Hudson-2 USB2" rev 0x39: apic 4 int 17
usb1 at ehci0: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "AMD EHCI root hub" rev 2.00/1.00 addr 1
ohci1 at pci0 dev 19 function 0 "AMD Hudson-2 USB" rev 0x39: apic 4 int 18, version 1.0, legacy support
ehci1 at pci0 dev 19 function 2 "AMD Hudson-2 USB2" rev 0x39: apic 4 int 17
usb2 at ehci1: USB revision 2.0
uhub2 at usb2 configuration 1 interface 0 "AMD EHCI root hub" rev 2.00/1.00 addr 1
piixpm0 at pci0 dev 20 function 0 "AMD Hudson-2 SMBus" rev 0x3a: SMBus disabled
azalia1 at pci0 dev 20 function 2 "AMD Hudson-2 HD Audio" rev 0x02: apic 4 int 16
azalia1: codecs: Realtek ALC282
audio0 at azalia1
pcib0 at pci0 dev 20 function 3 "AMD Hudson-2 LPC" rev 0x11
sdhc0 at pci0 dev 20 function 7 "AMD Bolton SD/MMC" rev 0x01: apic 4 int 16
sdhc0: SDHC 2.0, 50 MHz base clock
sdmmc1 at sdhc0: 4-bit, sd high-speed, mmc high-speed, dma
pchb2 at pci0 dev 24 function 0 "AMD 16h Link Cfg" rev 0x00
pchb3 at pci0 dev 24 function 1 "AMD 16h Address Map" rev 0x00
pchb4 at pci0 dev 24 function 2 "AMD 16h DRAM Cfg" rev 0x00
km0 at pci0 dev 24 function 3 "AMD 16h Misc Cfg" rev 0x00
pchb5 at pci0 dev 24 function 4 "AMD 16h CPU Power" rev 0x00
pchb6 at pci0 dev 24 function 5 vendor "AMD", unknown product 0x1535 rev 0x00
usb3 at ohci0: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "AMD OHCI root hub" rev 1.00/1.00 addr 1
usb4 at ohci1: USB revision 1.0
uhub4 at usb4 configuration 1 interface 0 "AMD OHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pms0: Synaptics touchpad, firmware 7.5, 0x1e0b1 0x240000 0x188f40 0xf00073 0xa2400
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: SVM/RVI
uvideo0 at uhub2 port 1 configuration 1 interface 0 "SunplusIT INC. HP Truevision HD" rev 2.00/1.05 addr 2
video0 at uvideo0
ugen0 at uhub3 port 4 "Broadcom Corp BCM20702A0" rev 2.00/1.12 addr 2
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
sd1 at scsibus3 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006>
sd1: 305245MB, 512 bytes/sector, 625141856 sectors
root on sd1a (cdfb4b96946677cc.a) swap on sd1b dump on sd1b
WARNING: clock gained 12 days
WARNING: CHECK AND RESET THE DATE!
radeondrm0: KABINI
radeondrm0: 1366x768, 32bpp
wsdisplay0 at radeondrm0 mux 1: console (std, vt100 emulation), using wskbd0
wsdisplay0: screen 1-5 added (std, vt100 emulation)


$ mount
/dev/sd1a on / type ffs (local, noatime, softdep)
/dev/sd1j on /ibs type ffs (local, noatime, nodev, nosuid, softdep)
/dev/sd1h on /home type ffs (local, noatime, nodev, nosuid, softdep)
/dev/sd1f on /tmp type ffs (local, noatime, nodev, nosuid, softdep)
/dev/sd1d on /usr type ffs (local, noatime, nodev, softdep)
/dev/sd1e on /usr/local type ffs (local, noatime, nodev, wxallowed, softdep)
/dev/sd1k on /v type ffs (local, noatime, nodev, nosuid, softdep)
/dev/sd1n on /v/1 type ffs (local, noatime, nodev, nosuid, softdep)
/dev/sd1o on /v/2 type ffs (local, noatime, nodev, nosuid, softdep)
/dev/sd1g on /var type ffs (local, noatime, nodev, nosuid, softdep)