ipsec-related panic?

Greetings all.

I have been playing around with GRE tunnel over IPSec. A couple of times
I have seen either of the endpoints panic about 5GB into a large
transfer using ssh between 2 nodes which live within the networks
connected by this tunnel. These transfers aside, there isn't much going
on in this tunnel but with a lot of activity this event is recurring.

Both endpoints are running OpenBSD 6.3 with all current patches applied
by syspatch and the hardware used is PCEngines APU2
(http://pcengines.ch/apu2c4.htm). I also tried with plain 6.3 (ie. no
patches) which was the original state of these endpoints.

Here is the info which I have been able to gather. I can't show you
'ipsecctl -sa' at the moment since the node which just panic'ed needs
somebody on the other end to power cycle it.

I can pursue getting more info if needed.

Any thoughts?

Thanks a lot!

Joao

# ifconfig


lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 32768
index 5 priority 0 llprio 3
groups: lo
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
inet 127.0.0.1 netmask 0xff000000
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:0d:b9:48:b8:f4
index 1 priority 0 llprio 3
groups: egress
media: Ethernet autoselect (1000baseT full-duplex)
status: active
inet 77.xx.yy.214 netmask 0xffffffc0 broadcast 77.........
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:0d:b9:48:b8:f5
index 2 priority 0 llprio 3
media: Ethernet autoselect (1000baseT full-duplex)
status: active
inet 10.22.0.32 netmask 0xfffffffe
em2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:0d:b9:48:b8:f6
index 3 priority 0 llprio 3
media: Ethernet autoselect (none)
status: no carrier
enc0: flags=41<UP,RUNNING>
index 4 priority 0 llprio 3
groups: enc
status: active
gre0: flags=9051<UP,POINTOPOINT,RUNNING,LINK0,MULTICAST> mtu 1476
index 6 priority 0 llprio 3
encap: vnetid none
groups: gre
tunnel: inet 77.xx.yy.214 -> 77.xx.yy.216 ttl 64 nodf
inet 172.17.0.1 --> 172.17.0.2 netmask 0xffffffff
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 32768
index 7 priority 0 llprio 3
groups: lo
inet 172.22.0.9 netmask 0xffffffff
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33136
index 8 priority 0 llprio 3
groups: pflog

# dmesg
OpenBSD 6.3 (GENERIC.MP) #10: Wed Aug 22 16:42:31 CEST 2018

root@syspatch-63-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4261076992 (4063MB)
avail mem = 4124860416 (3933MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xdffb7020 (7 entries)
bios0: vendor coreboot version "4.0.7" date 02/28/2017
bios0: PC Engines APU2
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S2 S3 S4 S5
acpi0: tables DSDT FACP SSDT APIC HEST SSDT SSDT HPET
acpi0: wakeup devices PWRB(S4) PBR4(S4) PBR5(S4) PBR6(S4) PBR7(S4)
PBR8(S4) UOH1(S3) UOH3(S3) UOH5(S3) XHC0(S4)
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD GX-412TC SOC, 998.28 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu0: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB
64b/line 16-way L2 cache
cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu0: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
acpitimer0: recalibrated TSC frequency 998131167 Hz
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: AMD GX-412TC SOC, 998.13 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu1: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB
64b/line 16-way L2 cache
cpu1: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu1: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: AMD GX-412TC SOC, 998.13 MHz
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu2: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB
64b/line 16-way L2 cache
cpu2: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu2: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: AMD GX-412TC SOC, 998.13 MHz
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu3: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB
64b/line 16-way L2 cache
cpu3: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu3: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 21, 24 pins
ioapic1 at mainbus0: apid 5 pa 0xfec20000, version 21, 32 pins
, remapped to apid 5
acpihpet0 at acpi0: 14318180 Hz
acpihpet0: recalibrated TSC frequency 998129868 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (PBR4)
acpiprt2 at acpi0: bus 1 (PBR5)
acpiprt3 at acpi0: bus 2 (PBR6)
acpiprt4 at acpi0: bus 3 (PBR7)
acpiprt5 at acpi0: bus -1 (PBR8)
acpicpu0 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpicpu1 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpicpu2 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpicpu3 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpibtn0 at acpi0: PWRB
cpu0: 998 MHz: speeds: 1000 800 600 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "AMD AMD64 16h Root Complex" rev 0x00
pchb1 at pci0 dev 2 function 0 "AMD AMD64 16h Host" rev 0x00
ppb0 at pci0 dev 2 function 2 "AMD AMD64 16h PCIE" rev 0x00: msi
pci1 at ppb0 bus 1
em0 at pci1 dev 0 function 0 "Intel I210" rev 0x03: msi, address
00:0d:b9:48:b8:f4
ppb1 at pci0 dev 2 function 3 "AMD AMD64 16h PCIE" rev 0x00: msi
pci2 at ppb1 bus 2
em1 at pci2 dev 0 function 0 "Intel I210" rev 0x03: msi, address
00:0d:b9:48:b8:f5
ppb2 at pci0 dev 2 function 4 "AMD AMD64 16h PCIE" rev 0x00: msi
pci3 at ppb2 bus 3
em2 at pci3 dev 0 function 0 "Intel I210" rev 0x03: msi, address
00:0d:b9:48:b8:f6
"AMD CCP" rev 0x00 at pci0 dev 8 function 0 not configured
xhci0 at pci0 dev 16 function 0 "AMD Bolton xHCI" rev 0x11: msi
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "AMD xHCI root hub" rev
3.00/1.00 addr 1
ahci0 at pci0 dev 17 function 0 "AMD Hudson-2 SATA" rev 0x40: apic 4 int
19, AHCI 1.3
ahci0: port 0: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, SATA SSD, SBFM> SCSI3 0/direct fixed
naa.5000000000000000
sd0: 15272MB, 512 bytes/sector, 31277232 sectors, thin
ehci0 at pci0 dev 19 function 0 "AMD Hudson-2 USB2" rev 0x39: apic 4 int 18
usb1 at ehci0: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "AMD EHCI root hub" rev
2.00/1.00 addr 1
piixpm0 at pci0 dev 20 function 0 "AMD Hudson-2 SMBus" rev 0x42: SMBus
disabled
pcib0 at pci0 dev 20 function 3 "AMD Hudson-2 LPC" rev 0x11
sdhc0 at pci0 dev 20 function 7 "AMD Bolton SD/MMC" rev 0x01: apic 4 int 16
sdhc0: SDHC 2.0, 50 MHz base clock
sdmmc0 at sdhc0: 4-bit, sd high-speed, mmc high-speed, dma
pchb2 at pci0 dev 24 function 0 "AMD AMD64 16h Link Cfg" rev 0x00
pchb3 at pci0 dev 24 function 1 "AMD AMD64 16h Address Map" rev 0x00
pchb4 at pci0 dev 24 function 2 "AMD AMD64 16h DRAM Cfg" rev 0x00
km0 at pci0 dev 24 function 3 "AMD AMD64 16h Misc Cfg" rev 0x00
pchb5 at pci0 dev 24 function 4 "AMD AMD64 16h CPU Power" rev 0x00
pchb6 at pci0 dev 24 function 5 "AMD AMD64 16h Misc Cfg" rev 0x00
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
wbsio0 at isa0 port 0x2e/2: NCT5104D rev 0x53
vmm0 at mainbus0: SVM/RVI
uhub2 at uhub1 port 1 configuration 1 interface 0 "Advanced Micro
Devices product 0x7900" rev 2.00/0.18 addr 2
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (fb09876ff93fa1fe.a) swap on sd0b dump on sd0b


# cat /etc/iked.conf


ikev2 passive esp from 77.xx.yy.214 to 77.xx.yy.216
ikev2 "tunnel" passive esp \
from 77.xx.yy.214/32 to 77.xx.yy.216/32 \
from 77.xx.yy.216/32 to 77.xx.yy.214/32 \
local 77.xx.yy.214 peer 77.xx.yy.216 \
srcid 77.xx.yy.214 dstid 77.xx.yy.216

# panic: Data modified on freelist: word 4 of object 0xffff800000b5f000
size 0x1000 previous type free (0x0 != 0xdeaf4152)

Stopped at db_enter+0x5: popq %rbp
TID PID UID PRFLAGS PFLAGS CPU COMMAND
*283793 97567 101 0x100010 0 2 iked
169307 26871 73 0x100010 0 0 syslogd
414157 17026 0 0x14000 0x200 1 crynlk
150338 42029 0 0x14000 0x200 3 crypto
db_enter() at db_enter+0x5
panic() at panic+0x129
malloc(0,16,ffff8000000cda40) at malloc+0x6e8
aesni_newsession(0,16) at aesni_newsession+0x5d
crypto_newsession(ffff8000221df328,ffff800000b4e558,ffff800000b4ec00) at
crypto
_newsession+0x10b
esp_init(ffff8000221df328,ffff800000b51a00,85e633f7b06deafe) at
esp_init+0x436
pfkeyv2_send(ffffff011cf58318,ffffff0009969c00,ffffff011cf58318) at
pfkeyv2_sen
d+0x18ef
pfkeyv2_output(ffff800000415840,2d,7,ffffffff81aa0960) at
pfkeyv2_output+0x6d
raw_usrreq(1ed0,0,0,ffffff0009969c00,ffff8000221df4f0,ffffff0009969c00)
at raw_
usrreq+0x13c
sosend(130,0,ffffff011d1d9820,e,ffffff011d1d97f0,85e633f7b06deafe) at
sosend+0x
345
dofilewritev(ffff8000221728f8,ffff8000221df650,ffffff011d1d97f0,3,ffff8000221df
660,85e633f7b06deafe) at dofilewritev+0x245
sys_writev(790,ffff8000221728f8,0) at sys_writev+0x75
syscall() at syscall+0x279
--- syscall (number 121) ---
end of kernel
end trace frame: 0x7f7fffff7b70, count: 2
0x6948f29f6ba:
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.

===============================================================

ddb{2}> mach ddbcpu 0
Stopped at x86_ipi_db+0x5: popq %rbp
x86_ipi_db(ffff800000082000) at x86_ipi_db+0x5
x86_ipi_handler() at x86_ipi_handler+0x6a
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1f
--- interrupt ---
_kernel_lock(790,ffff800022136d60) at _kernel_lock+0x82
Xsoftclock() at Xsoftclock+0x1f
--- interrupt ---
end of kernel
end trace frame: 0x4, count: 10
0x8:
ddb{0}> trace
x86_ipi_db(ffff800000082000) at x86_ipi_db+0x5
x86_ipi_handler() at x86_ipi_handler+0x6a
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1f
--- interrupt ---
_kernel_lock(790,ffff800022136d60) at _kernel_lock+0x82
Xsoftclock() at Xsoftclock+0x1f
--- interrupt ---
end of kernel
end trace frame: 0x4, count: -5
0x8:

===============================================================

ddb{0}> mach ddbcpu 1
Stopped at x86_ipi_db+0x5: popq %rbp
x86_ipi_db(0) at x86_ipi_db+0x5
x86_ipi_handler() at x86_ipi_handler+0x6a
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1f
--- interrupt ---
msleep(ffff800000086898,ffff800000086880,ffffffff816957f0,ffff8000220f1780,0)
a
t msleep+0x10a
taskq_next_work(ffffffff816957f0,ffff800000086880,ffff800000086880) at
taskq_ne
xt_work+0x48
taskq_thread(0) at taskq_thread+0x8a
end trace frame: 0x0, count: 9
ddb{1}> trace
x86_ipi_db(0) at x86_ipi_db+0x5
x86_ipi_handler() at x86_ipi_handler+0x6a
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1f
--- interrupt ---
msleep(ffff800000086898,ffff800000086880,ffffffff816957f0,ffff8000220f1780,0)
a
t msleep+0x10a
taskq_next_work(ffffffff816957f0,ffff800000086880,ffff800000086880) at
taskq_ne
xt_work+0x48
taskq_thread(0) at taskq_thread+0x8a
end trace frame: 0x0, count: -6

===============================================================

ddb{1}> mach ddbcpu 2
Stopped at db_enter+0x5: popq %rbp
db_enter() at db_enter+0x5
panic() at panic+0x129
malloc(0,16,ffff8000000cda40) at malloc+0x6e8
aesni_newsession(0,16) at aesni_newsession+0x5d
crypto_newsession(ffff8000221df328,ffff800000b4e558,ffff800000b4ec00) at
crypto
_newsession+0x10b
esp_init(ffff8000221df328,ffff800000b51a00,85e633f7b06deafe) at
esp_init+0x436
pfkeyv2_send(ffffff011cf58318,ffffff0009969c00,ffffff011cf58318) at
pfkeyv2_sen
d+0x18ef
pfkeyv2_output(ffff800000415840,2d,7,ffffffff81aa0960) at
pfkeyv2_output+0x6d
raw_usrreq(1ed0,0,0,ffffff0009969c00,ffff8000221df4f0,ffffff0009969c00)
at raw_
usrreq+0x13c
sosend(130,0,ffffff011d1d9820,e,ffffff011d1d97f0,85e633f7b06deafe) at
sosend+0x
345
dofilewritev(ffff8000221728f8,ffff8000221df650,ffffff011d1d97f0,3,ffff8000221df
660,85e633f7b06deafe) at dofilewritev+0x245
sys_writev(790,ffff8000221728f8,0) at sys_writev+0x75
syscall() at syscall+0x279
--- syscall (number 121) ---
end of kernel
end trace frame: 0x7f7fffff7b70, count: 2
0x6948f29f6ba:
ddb{2}> trace
db_enter() at db_enter+0x5
panic() at panic+0x129
malloc(0,16,ffff8000000cda40) at malloc+0x6e8
aesni_newsession(0,16) at aesni_newsession+0x5d
crypto_newsession(ffff8000221df328,ffff800000b4e558,ffff800000b4ec00) at
crypto
_newsession+0x10b
esp_init(ffff8000221df328,ffff800000b51a00,85e633f7b06deafe) at
esp_init+0x436
pfkeyv2_send(ffffff011cf58318,ffffff0009969c00,ffffff011cf58318) at
pfkeyv2_sen
d+0x18ef
pfkeyv2_output(ffff800000415840,2d,7,ffffffff81aa0960) at
pfkeyv2_output+0x6d
raw_usrreq(1ed0,0,0,ffffff0009969c00,ffff8000221df4f0,ffffff0009969c00)
at raw_
usrreq+0x13c
sosend(130,0,ffffff011d1d9820,e,ffffff011d1d97f0,85e633f7b06deafe) at
sosend+0x
345
dofilewritev(ffff8000221728f8,ffff8000221df650,ffffff011d1d97f0,3,ffff8000221df
660,85e633f7b06deafe) at dofilewritev+0x245
sys_writev(790,ffff8000221728f8,0) at sys_writev+0x75
syscall() at syscall+0x279
--- syscall (number 121) ---
end of kernel
end trace frame: 0x7f7fffff7b70, count: -13
0x6948f29f6ba:

===============================================================


ddb{2}> mach ddbcpu 3
Stopped at x86_ipi_db+0x5: popq %rbp
x86_ipi_db(ffffffff813e7ab5) at x86_ipi_db+0x5
x86_ipi_handler() at x86_ipi_handler+0x6a
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1f
--- interrupt ---
__mtx_enter(b43) at __mtx_enter+0x31
free(ffffff00cd7e2100,ffffff011b092ca8,ffff8000004d5c00) at free+0x198
esp_output_cb(ffffffff816957f0) at esp_output_cb+0x134
taskq_thread(0) at taskq_thread+0x67
end trace frame: 0x0, count: 8
ddb{3}> trace
x86_ipi_db(ffffffff813e7ab5) at x86_ipi_db+0x5
x86_ipi_handler() at x86_ipi_handler+0x6a
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1f
--- interrupt ---
__mtx_enter(b43) at __mtx_enter+0x31
free(ffffff00cd7e2100,ffffff011b092ca8,ffff8000004d5c00) at free+0x198
esp_output_cb(ffffffff816957f0) at esp_output_cb+0x134
taskq_thread(0) at taskq_thread+0x67
end trace frame: 0x0, count: -7

===============================================================

ddb{3}> ps
PID TID PPID UID S FLAGS WAIT COMMAND
41265 305068 26658 0 3 0x100083 ttyin ksh
26658 373858 88560 1000 3 0x10008b pause ksh
88560 248020 15756 1000 3 0x90 select sshd
15756 419833 74060 0 3 0x92 poll sshd
9353 463803 60940 0 3 0x100083 ttyin ksh
60940 98127 1 1000 3 0x10008b pause ksh
65515 139304 72235 75 3 0x100092 poll bgpd
82387 504706 72235 75 3 0x100092 poll bgpd
72235 518323 1 0 3 0x80 poll bgpd
34819 33384 1 0 3 0x100098 poll cron
99537 63369 1 99 3 0x100090 poll sndiod
66516 83150 1 110 3 0x100090 poll sndiod
27526 70856 31875 95 3 0x100092 kqread smtpd
19585 148620 31875 103 3 0x100092 kqread smtpd
87636 498579 31875 95 3 0x100092 kqread smtpd
52954 114205 31875 95 3 0x100092 kqread smtpd
20901 44536 31875 95 3 0x100092 kqread smtpd
24606 155188 31875 95 3 0x100092 kqread smtpd
31875 385445 1 0 3 0x100080 kqread smtpd
74060 209053 1 0 3 0x80 select sshd
97567 283793 96516 101 7 0x100010 iked
53040 418919 96516 101 3 0x100090 kqread iked
6049 124916 96516 101 3 0x100090 kqread iked
96516 6627 1 0 3 0x100080 kqread iked
47727 513154 56740 83 3 0x100092 poll ntpd
56740 244325 55902 83 3 0x100092 poll ntpd
55902 338853 1 0 3 0x100080 poll ntpd
93590 79542 53651 74 3 0x100092 bpf pflogd
53651 472065 1 0 3 0x80 netio pflogd
26871 169307 1879 73 7 0x100010 syslogd
1879 393274 1 0 3 0x100082 netio syslogd
64224 155424 37136 115 3 0x100092 kqread slaacd
94862 13687 37136 115 3 0x100092 kqread slaacd
37136 380932 1 0 3 0x80 kqread slaacd
28370 275217 0 0 3 0x14200 pgzero zerothread
67512 25938 0 0 3 0x14200 aiodoned aiodoned
43551 309694 0 0 3 0x14200 syncer update
37782 284291 0 0 3 0x14200 cleaner cleaner
23716 460330 0 0 3 0x14200 reaper reaper
6888 521664 0 0 3 0x14200 pgdaemon pagedaemon
17026 414157 0 0 7 0x14200 crynlk
*42029 150338 0 0 7 0x14200 crypto
64057 505108 0 0 3 0x14200 bored sensors
56497 57629 0 0 3 0x14200 mmctsk sdmmc0
12289 153527 0 0 3 0x14200 usbtsk usbtask
57726 120118 0 0 3 0x14200 usbatsk usbatsk
99464 463119 0 0 3 0x40014200 acpi0 acpi0
92349 432456 0 0 3 0x40014200 idle3
65420 502413 0 0 3 0x40014200 idle2
76924 62177 0 0 3 0x40014200 idle1
21964 164284 0 0 2 0x14200 softnet
91618 137985 0 0 3 0x14200 bored systqmp
67418 278845 0 0 3 0x14200 bored systq
2340 417013 0 0 3 0x40014200 bored softclock
62767 364056 0 0 3 0x40014200 idle0
1 340309 0 0 3 0x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper