Sunday, April 30, 2023

7.2 panic and "reorder_kernel: failed" ...

Hi All,

Our 7.2 system just paniced again in pmap_page_remove / uvm_fault:
> ddb{1}> show panic
> *cpu1: uvm_fault(0xfffffd818b0ca560, 0x7f817ca74cb0, 0, 2) -> e
> ddb{1}> trace
> pmap_page_remove(fffffd8109c56480) at pmap_page_remove+0x21d
> uvm_anfree_list(fffffd804a0e7e40,ffff800022eab518) at uvm_anfree_list+0x56
> amap_wipeout(fffffd80553d12e0) at amap_wipeout+0x113
> uvm_unmap_detach(ffff800022eab5d8,0) at uvm_unmap_detach+0x6d
> sys_munmap(ffff800022720010,ffff800022eab640,ffff800022eab6a0) at sys_munmap+0x113
> syscall(ffff800022eab710) at syscall+0x35f
> Xsyscall() at Xsyscall+0x128
> end of kernel
> end trace frame: 0x7f7ffffdcd10, count: -7

(See also bug report from 24.4, subject: "kernel panic in pmap_page_remove")

After running fsck manually to clean one of the filesystems I did an
additional reboot, just to be sure the system would/could come up
cleanly.

I noticed this message on the console, seemingly as the system was
shutting down:
> stopping package daemons: nginx slowcgi postfix cyrus_imapd(killed) amavisd clamd sshguardreorder_kernel: failed -- see /usr/share/relink/kernel/GENERIC.MP/relink.log

That relink.log file looks like this:
> root:[~]# ls -ltr /usr/share/relink/kernel/GENERIC.MP/relink.log
> -rw-r--r-- 1 root wheel 142 Apr 30 14:29 /usr/share/relink/kernel/GENERIC.MP/relink.log

> root:[~]# cat /usr/share/relink/kernel/GENERIC.MP/relink.log
> (SHA256) /bsd: OK
> LD="ld" sh makegap.sh 0xcccccccc gapdummy.o
> ld -T ld.script -X --warn-common -nopie -o newbsd ${SYSTEM_HEAD} vers.o ${OBJS}
> root:[~]#

What might that mean? Is it significant?

This is a virtualised OpenBSD instance running on QEMU / Debian Linux.

I've included some additional info, regarding the panic, output from ddb,
below ... if that is of interest.

Cheers,
Robb.


ddb{0}> show uvm
Current UVM status:
pagesize=4096 (0x1000), pagemask=0xfff, pageshift=121519007 VM pages: 615597 active, 54738 inactive, 1 wired, 299836 free (77933 zero)
min 10% (25) anon, 10% (25) vnode, 5% (12) vtext
freemin=50633, free-target=67510, inactive-target=0, wired-max=506335
faults=147405838, traps=149196391, intrs=4821202, ctxswitch=33921314 fpuswitch=0
softint=46722165, syscalls=281857273, kmapent=11
fault counts:
noram=0, noanon=0, noamap=0, pgwait=0, pgrele=0
ok relocks(total)=238350(239289), anget(retries)=19653204(0), amapcopy=11109730
neighbor anon/obj pg=5361492/21511643, gets(lock/unlock)=9626010/239342
cases: anon=17935690, anoncow=1717514, obj=5849938, prcopy=3775080, przero=118127605
daemon and swap counts:
woke=0, revs=0, scans=0, obscans=0, anscans=0
busy=0, freed=0, reactivate=0, deactivate=0
pageouts=0, pending=0, nswget=0
nswapdev=1
swpages=526128, swpginuse=0, swpgonly=0 paging=0
kernel pointers:
objs(kern)=0xffffffff822e7588


ddb{0}> ps
PID TID PPID UID S FLAGS WAIT COMMAND
85463 376479 19056 543 3 0x8a kqread lmtpd
40689 333687 63656 507 3 0x92 kqread lmtp
20679 171977 63656 507 3 0x1000092 kqread cleanup
62580 393095 63656 507 3 0x92 kqread smtpd
94055 484524 63656 507 3 0x1000092 kqread trivial-rewrite
63319 120941 63656 507 3 0x1000092 kqread anvil
18813 417198 63656 507 3 0x92 kqread smtpd
25010 114571 19056 543 3 0x82 lockf imapd
53158 375953 19056 543 3 0x8a kqread imapd
68381 324871 63656 507 3 0x92 kqread tlsproxy
63605 58831 19056 543 3 0x8a kqread imapd
52360 292291 19056 543 3 0x8a kqread imapd
99886 433302 63656 507 3 0x92 lockf dnsblog
76801 2093 19056 543 7 0x2 lmtpd
25035 175379 17871 530 3 0x90 kqread perl
6203 299107 17871 530 3 0x90 lockf perl
71793 709 19056 543 3 0x8a kqread imapd
79965 216521 19056 543 3 0x8a kqread imapd
62342 450749 19056 543 3 0x8a kqread imapd
13176 514534 63656 507 3 0x92 kqread dnsblog
27955 360632 63656 507 3 0x92 lockf dnsblog
69483 431512 63656 507 3 0x92 kqread postscreen
10002 58404 19056 543 3 0x8a kqread imapd
46558 211092 19056 543 3 0x8a kqread imapd
46541 45982 19056 543 3 0x8a kqread imapd
14453 401009 19056 543 3 0x8a kqread imapd
36547 92359 63656 507 3 0x1000092 kqread pickup
68439 377693 17871 530 3 0x90 lockf perl
20312 193348 17871 530 3 0x90 lockf perl
29531 309738 17871 530 3 0x90 lockf perl
47097 134056 19056 543 3 0x8a kqread imapd
57064 215926 19056 543 3 0x8a kqread imapd
43454 515476 19056 543 3 0x8a kqread imapd
52531 107758 19056 543 3 0x8a kqread imapd
33801 57425 19056 543 3 0x8a kqread imapd
49846 13973 19056 543 3 0x8a kqread imapd
42145 176266 93902 0 3 0x8b kqread bash
93902 9053 54833 0 3 0x9a kqread sshd
34982 228903 60614 0 3 0x8b kqread bash
60614 347476 54833 0 3 0x9a kqread sshd
11670 117317 1 530 3 0x90 kqread clamd
11670 426746 1 530 3 0x4000090 kqread clamd
11670 235947 1 530 3 0x4000090 fsleep clamd
11670 320246 1 530 3 0x4000090 fsleep clamd
11670 295866 1 530 3 0x4000090 fsleep clamd
54837 466133 19056 543 3 0x8a kqread imapd
48061 478798 44023 0 3 0x8b kqread bash
44023 492165 54833 0 3 0x9a kqread sshd
52489 349099 19056 543 3 0x8a kqread imapd
52311 224899 19056 543 3 0x8a kqread imapd
74095 249565 63656 507 3 0x1000092 kqread tlsmgr
72026 255937 1 0 3 0x8b kqread bash
64677 248794 1 0 3 0x100098 kqread cron
6095 91613 31987 67 3 0x1000090 kqread nginx
31987 267143 1 0 3 0x88 sigsusp nginx
27947 82904 1 67 3 0x1100090 kqread slowcgi
67265 270589 63656 507 3 0x1000092 kqread qmgr
63656 241447 1 0 3 0x90 kqread master
*63721 337486 19056 543 7 0x2 idled
19056 272787 1 543 3 0x98 kqread master
17871 267458 1 530 3 0x90 kqsel perl
13800 457141 43780 0 3 0x100083 piperd sshg-parser
98864 274735 23854 0 3 0x100083 kqread tail
32104 460586 45850 0 3 0x100083 piperd sh
39891 29692 45850 0 3 0x100083 piperd sshg-blocker
39891 1450 45850 0 3 0x4100083 nanoslp sshg-blocker
43780 34759 45850 0 3 0x100089 sigsusp sh
23854 131943 45850 0 3 0x100089 sigsusp sh
45850 174555 1 0 3 0x10008b sigsusp sh
46941 358546 1 741 3 0x98 sigwait named
46941 145804 1 741 3 0x4000090 kqread named
46941 84705 1 741 3 0x4000090 kqread named
46941 207443 1 741 3 0x4000090 fsleep named
46941 448133 1 741 3 0x4000090 fsleep named
46941 245427 1 741 3 0x4000090 fsleep named
14118 220576 1 67 3 0x1100090 kqread slowcgi
54833 350167 1 0 3 0x88 kqread sshd
54504 485230 1 0 3 0x100080 kqread ntpd
20736 84451 96091 83 3 0x100092 kqread ntpd
96091 285344 1 83 3 0x1100092 kqread ntpd
63121 219625 8516 74 3 0x1100092 bpf pflogd
8516 487239 1 0 3 0x80 netio pflogd
67801 164031 49750 73 3 0x1100090 kqread syslogd
49750 176631 1 0 3 0x100082 netio syslogd
1069 411124 1 0 3 0x100080 kqread resolvd
42169 52342 78644 77 3 0x100092 kqread dhcpleased
90717 493864 78644 77 3 0x100092 kqread dhcpleased
78644 333452 1 0 3 0x80 kqread dhcpleased
52973 167549 1 0 3 0x80 mfsidl mount_mfs
91972 114073 1 0 3 0x80 mfsidl mount_mfs
55817 446 0 0 3 0x14200 bored smr
49977 452250 0 0 3 0x14200 pgzero zerothread
29252 282605 0 0 3 0x14200 aiodoned aiodoned
80176 366871 0 0 3 0x14200 syncer update
96546 85103 0 0 3 0x14200 cleaner cleaner
75419 458137 0 0 3 0x14200 reaper reaper
25092 357015 0 0 3 0x14200 pgdaemon pagedaemon
13466 268489 0 0 3 0x14200 bored viomb
64440 310283 0 0 3 0x14200 usbtsk usbtask
62780 387614 0 0 3 0x14200 usbatsk usbatsk
33472 208000 0 0 3 0x40014200 acpi0 acpi0
25600 152199 0 0 3 0x40014200 idle1
7027 135848 0 0 3 0x14200 bored softnet
39282 255203 0 0 3 0x14200 bored softnet
45681 244426 0 0 3 0x14200 bored softnet
63633 94628 0 0 3 0x14200 bored softnet
42667 106196 0 0 3 0x14200 bored systqmp
60443 521897 0 0 3 0x14200 bored systq
49351 202720 0 0 3 0x40014200 bored softclock
94221 63107 0 0 3 0x40014200 idle0
1 60670 0 0 3 0x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper

No comments:

Post a Comment