On 16.04.2022 20:44, Noth wrote:
>Hi,
Hi,
> I built the new 7.1 packages for Zabbix and PostgreSQL, and upgraded
>my monitoring VM. To my horror the zabbix_server process stops
>graphing after a couple of hours of uptime, with the housekeeper and
>history syncer processes at over 80% cpu usage. PostgreSQL shows
>INSERT processes stuck at 80% too, and restarting zabbix_server hangs,
>leaving zombie processes. I was using 2 vcpus and 2G of RAM. Feeling
>that the login.conf limits might be the problem I uped them by quite a
>bit:
The zabbix_server process won't eat up your system normally...it's more
database related.
1) how big is your database? (important, as you don't have much memory
for a busy PostgreSQL database)
2) did you make changes to /var/postgresql/data/postgresql.conf?
(work_mem, shared_buffers, effective_cache_size, random_page_cost)
2) how many items do you monitor?
3) did you always follow the Zabbix upgrade guide (especially 5->6)?
4) how do your Zabbix server graphs "data gathering", "internal",
"data handling" look like before the problem starts? Any high
spikes?
5) have you already tried with a new database instance, i.e. move
/var/postgresql/data to /var/postgresql/data_old and start with
Zabbix 6.0.3 from scratch to see, if the database issues start
immediately even when almost empty?
>postgresql:\
>- :openfiles=1024:\
>+ :openfiles=4096:\
> :tc=daemon:
> :datasize-max=2048M:\
> :datasize-cur=2048M:\
In almost all environments i've dealt with so far, you didn't need a
postgresql login class at all.
-Mark
>I also uped sysctls:
>
>-kern.seminfo.semmns=2048
>-kern.shminfo.shmall=1024512
>+kern.seminfo.semmns=4096
>+kern.shminfo.shmall=1572864
>
>
>This helped quite a bit, the graphing at least doesn't stop. Yet the housekeeper & history sync processes are still stuck continuously at over 80%. I even doubled the RAM to 4G and the vcpus to 4. Yet this is what top shows:
>
>CPU0 states: 82.4% user, 0.0% nice, 14.7% sys, 1.0% spin, 0.0% intr, 2.0% idle
>CPU1 states: 79.4% user, 0.0% nice, 9.8% sys, 0.0% spin, 0.0% intr, 10.8% idle
>CPU2 states: 80.4% user, 0.0% nice, 9.8% sys, 0.0% spin, 0.0% intr, 9.8% idle
>CPU3 states: 94.1% user, 0.0% nice, 4.9% sys, 0.0% spin, 0.0% intr, 1.0% idle
>Memory: Real: 738M/2131M act/tot Free: 1796M Cache: 1091M Swap: 0K/2055M
>
> PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
>48835 _postgre 64 0 307M 247M onproc/0 - 23.4H 98.24% postgres: zabbix zabbix [local] SELECT
>42079 _postgre 64 0 307M 245M run/2 - 25.3H 98.10% postgres: zabbix zabbix [local] SELECT
> 3630 _postgre 63 0 308M 282M onproc/1 - 21.2H 82.42% postgres: zabbix zabbix [local] DELETE
>12622 _postgre 60 0 307M 127M onproc/2 - 20.8H 79.10% postgres: zabbix zabbix [local] INSERT
>
>None of this behaviour happened with versions 2.x through to 5.x. Yes
>I've followed the pkg-readme for both PostgreSQL and Zabbix. Maybe
>these need adjusting? I'm at a loss on what I need to tune to make
>everything go back to being running with low CPU usage. I am fully
>aware 6.x now collects data points of a gazillion more things (so many
>entries with just $1 as the name now, I'm not sure what's going on
>there).
>
>Hopefully some of you have an idea, cheers,
>
>Noth
>
>OpenBSD 7.1 (GENERIC.MP) #1: Wed Apr 6 18:48:24 CEST 2022
>root@builder2.nineinchnetworks.ch:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>real mem = 4255670272 (4058MB)
>avail mem = 4109398016 (3919MB)
>random: good seed from bootblocks
>mpath0 at root
>scsibus0 at mpath0: 256 targets
>mainbus0 at root
>bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xbfbcb000 (13 entries)
>bios0: vendor BHYVE version "13.0" date 11/10/2020
>bios0: FreeBSD BHYVE
>acpi0 at bios0: ACPI 4.0
>acpi0: sleep states S5
>acpi0: tables DSDT FACP HPET APIC MCFG SPCR
>acpi0: wakeup devices
>acpitimer0 at acpi0: 3579545 Hz, 32 bits
>acpihpet0 at acpi0: 16777216 Hz
>acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
>cpu0 at mainbus0: apid 0 (boot processor)
>cpu0: Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz, 3408.57 MHz, 06-9e-0d
>cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,RDSEED,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
>cpu0: 256KB 64b/line 8-way L2 cache
>cpu0: smt 0, core 0, package 0
>mtrr: CPU supports MTRRs but not enabled by BIOS
>cpu0: apic clock running at 134MHz
>cpu1 at mainbus0: apid 1 (application processor)
>cpu1: Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz, 3413.28 MHz, 06-9e-0d
>cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,RDSEED,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
>cpu1: 256KB 64b/line 8-way L2 cache
>cpu1: smt 0, core 0, package 1
>cpu2 at mainbus0: apid 2 (application processor)
>cpu2: Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz, 3411.94 MHz, 06-9e-0d
>cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,RDSEED,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
>cpu2: 256KB 64b/line 8-way L2 cache
>cpu2: smt 0, core 0, package 2
>cpu3 at mainbus0: apid 3 (application processor)
>cpu3: Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz, 3419.69 MHz, 06-9e-0d
>cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,RDSEED,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
>cpu3: 256KB 64b/line 8-way L2 cache
>cpu3: smt 0, core 0, package 3
>ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 11, 32 pins, remapped
>acpimcfg0 at acpi0
>acpimcfg0: addr 0xe0000000, bus 0-255
>acpiprt0 at acpi0: bus 0 (PC00)
>acpipci0 at acpi0 PC00
>com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
>com0: console
>com1 at acpi0 COM2 addr 0x2f8/0x8 irq 3: ns16550a, 16 byte fifo
>acpicmos0 at acpi0
>cpu0: using VERW MDS workaround
>pvbus0 at mainbus0: bhyve
>pci0 at mainbus0 bus 0
>pchb0 at pci0 dev 0 function 0 unknown vendor 0x1275 product 0x1275 rev 0x00
>ahci0 at pci0 dev 3 function 0 "Intel 82801H AHCI" rev 0x00: msi, AHCI 1.3
>ahci0: port 0: 6.0Gb/s
>scsibus1 at ahci0: 32 targets
>cd0 at scsibus1 targ 0 lun 0: <BHYVE, BHYVE DVD-ROM, 001> removable
>virtio0 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
>vioblk0 at virtio0
>scsibus2 at vioblk0: 1 targets
>sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, >
>sd0: 204800MB, 512 bytes/sector, 419430400 sectors
>virtio0: msix shared
>virtio1 at pci0 dev 5 function 0 "Qumranet Virtio Storage" rev 0x00
>vioblk1 at virtio1
>scsibus3 at vioblk1: 1 targets
>sd1 at scsibus3 targ 0 lun 0: <VirtIO, Block Device, >
>sd1: 664MB, 512 bytes/sector, 1360832 sectors
>virtio1: msix shared
>virtio2 at pci0 dev 6 function 0 "Qumranet Virtio Network" rev 0x00
>vio0 at virtio2: address 00:a0:98:16:bb:2c
>virtio2: msix shared
>pcib0 at pci0 dev 31 function 0 "Intel 82371SB ISA" rev 0x00
>isa0 at pcib0
>isadma0 at isa0
>com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo
>pckbc0 at isa0 port 0x60/5 irq 1 irq 12
>pckbd0 at pckbc0 (kbd slot)
>wskbd0 at pckbd0 mux 1
>pms0 at pckbc0 (aux slot)
>wsmouse0 at pms0 mux 0
>efifb at mainbus0 not configured
>vscsi0 at root
>scsibus4 at vscsi0: 256 targets
>softraid0 at root
>scsibus5 at softraid0: 256 targets
>root on sd0a (2b7e2c838e3db383.a) swap on sd0b dump on sd0b
>
--
Mark Patruck ( mark at wrapped.cx )
GPG key 0xF2865E51 / 187F F6D3 EE04 1DCE 1C74 F644 0D3C F66F F286 5E51
https://www.wrapped.cx
No comments:
Post a Comment