On 19.04.2022 10:30, Mark Patruck wrote:
>On 16.04.2022 20:44, Noth wrote:
>>Hi,
>
>Hi,
>
>> I built the new 7.1 packages for Zabbix and PostgreSQL, and
>>upgraded my monitoring VM. To my horror the zabbix_server process
>>stops graphing after a couple of hours of uptime, with the
>>housekeeper and history syncer processes at over 80% cpu usage.
>>PostgreSQL shows INSERT processes stuck at 80% too, and restarting
>>zabbix_server hangs, leaving zombie processes. I was using 2 vcpus
>>and 2G of RAM. Feeling that the login.conf limits might be the
>>problem I uped them by quite a bit:
>
>The zabbix_server process won't eat up your system normally...it's more
>database related.
>
>1) how big is your database? (important, as you don't have much memory
> for a busy PostgreSQL database)
>
>2) did you make changes to /var/postgresql/data/postgresql.conf?
> (work_mem, shared_buffers, effective_cache_size, random_page_cost)
>
>2) how many items do you monitor?
>
>3) did you always follow the Zabbix upgrade guide (especially 5->6)?
>
>4) how do your Zabbix server graphs "data gathering", "internal",
> "data handling" look like before the problem starts? Any high
> spikes?
>
>5) have you already tried with a new database instance, i.e. move
> /var/postgresql/data to /var/postgresql/data_old and start with
> Zabbix 6.0.3 from scratch to see, if the database issues start
> immediately even when almost empty?
>
>>postgresql:\
>>- :openfiles=1024:\
>>+ :openfiles=4096:\
>> :tc=daemon:
>> :datasize-max=2048M:\
>> :datasize-cur=2048M:\
>
>In almost all environments i've dealt with so far, you didn't need a
>postgresql login class at all.
^^^^
Forget about that...i had the zabbix_server login class in mind.
> -Mark
>
>
>>I also uped sysctls:
>>
>>-kern.seminfo.semmns=2048
>>-kern.shminfo.shmall=1024512
>>+kern.seminfo.semmns=4096
>>+kern.shminfo.shmall=1572864
>>
>>
>>This helped quite a bit, the graphing at least doesn't stop. Yet the housekeeper & history sync processes are still stuck continuously at over 80%. I even doubled the RAM to 4G and the vcpus to 4. Yet this is what top shows:
>>
>>CPU0 states: 82.4% user, 0.0% nice, 14.7% sys, 1.0% spin, 0.0% intr, 2.0% idle
>>CPU1 states: 79.4% user, 0.0% nice, 9.8% sys, 0.0% spin, 0.0% intr, 10.8% idle
>>CPU2 states: 80.4% user, 0.0% nice, 9.8% sys, 0.0% spin, 0.0% intr, 9.8% idle
>>CPU3 states: 94.1% user, 0.0% nice, 4.9% sys, 0.0% spin, 0.0% intr, 1.0% idle
>>Memory: Real: 738M/2131M act/tot Free: 1796M Cache: 1091M Swap: 0K/2055M
>>
>> PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
>>48835 _postgre 64 0 307M 247M onproc/0 - 23.4H 98.24% postgres: zabbix zabbix [local] SELECT
>>42079 _postgre 64 0 307M 245M run/2 - 25.3H 98.10% postgres: zabbix zabbix [local] SELECT
>>3630 _postgre 63 0 308M 282M onproc/1 - 21.2H 82.42% postgres: zabbix zabbix [local] DELETE
>>12622 _postgre 60 0 307M 127M onproc/2 - 20.8H 79.10% postgres: zabbix zabbix [local] INSERT
>>
>>None of this behaviour happened with versions 2.x through to 5.x.
>>Yes I've followed the pkg-readme for both PostgreSQL and Zabbix.
>>Maybe these need adjusting? I'm at a loss on what I need to tune to
>>make everything go back to being running with low CPU usage. I am
>>fully aware 6.x now collects data points of a gazillion more things
>>(so many entries with just $1 as the name now, I'm not sure what's
>>going on there).
>>
>>Hopefully some of you have an idea, cheers,
>>
>>Noth
>>
>>OpenBSD 7.1 (GENERIC.MP) #1: Wed Apr 6 18:48:24 CEST 2022
>>root@builder2.nineinchnetworks.ch:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>>real mem = 4255670272 (4058MB)
>>avail mem = 4109398016 (3919MB)
>>random: good seed from bootblocks
>>mpath0 at root
>>scsibus0 at mpath0: 256 targets
>>mainbus0 at root
>>bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xbfbcb000 (13 entries)
>>bios0: vendor BHYVE version "13.0" date 11/10/2020
>>bios0: FreeBSD BHYVE
>>acpi0 at bios0: ACPI 4.0
>>acpi0: sleep states S5
>>acpi0: tables DSDT FACP HPET APIC MCFG SPCR
>>acpi0: wakeup devices
>>acpitimer0 at acpi0: 3579545 Hz, 32 bits
>>acpihpet0 at acpi0: 16777216 Hz
>>acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
>>cpu0 at mainbus0: apid 0 (boot processor)
>>cpu0: Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz, 3408.57 MHz, 06-9e-0d
>>cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,RDSEED,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
>>cpu0: 256KB 64b/line 8-way L2 cache
>>cpu0: smt 0, core 0, package 0
>>mtrr: CPU supports MTRRs but not enabled by BIOS
>>cpu0: apic clock running at 134MHz
>>cpu1 at mainbus0: apid 1 (application processor)
>>cpu1: Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz, 3413.28 MHz, 06-9e-0d
>>cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,RDSEED,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
>>cpu1: 256KB 64b/line 8-way L2 cache
>>cpu1: smt 0, core 0, package 1
>>cpu2 at mainbus0: apid 2 (application processor)
>>cpu2: Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz, 3411.94 MHz, 06-9e-0d
>>cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,RDSEED,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
>>cpu2: 256KB 64b/line 8-way L2 cache
>>cpu2: smt 0, core 0, package 2
>>cpu3 at mainbus0: apid 3 (application processor)
>>cpu3: Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz, 3419.69 MHz, 06-9e-0d
>>cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,RDSEED,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
>>cpu3: 256KB 64b/line 8-way L2 cache
>>cpu3: smt 0, core 0, package 3
>>ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 11, 32 pins, remapped
>>acpimcfg0 at acpi0
>>acpimcfg0: addr 0xe0000000, bus 0-255
>>acpiprt0 at acpi0: bus 0 (PC00)
>>acpipci0 at acpi0 PC00
>>com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
>>com0: console
>>com1 at acpi0 COM2 addr 0x2f8/0x8 irq 3: ns16550a, 16 byte fifo
>>acpicmos0 at acpi0
>>cpu0: using VERW MDS workaround
>>pvbus0 at mainbus0: bhyve
>>pci0 at mainbus0 bus 0
>>pchb0 at pci0 dev 0 function 0 unknown vendor 0x1275 product 0x1275 rev 0x00
>>ahci0 at pci0 dev 3 function 0 "Intel 82801H AHCI" rev 0x00: msi, AHCI 1.3
>>ahci0: port 0: 6.0Gb/s
>>scsibus1 at ahci0: 32 targets
>>cd0 at scsibus1 targ 0 lun 0: <BHYVE, BHYVE DVD-ROM, 001> removable
>>virtio0 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
>>vioblk0 at virtio0
>>scsibus2 at vioblk0: 1 targets
>>sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, >
>>sd0: 204800MB, 512 bytes/sector, 419430400 sectors
>>virtio0: msix shared
>>virtio1 at pci0 dev 5 function 0 "Qumranet Virtio Storage" rev 0x00
>>vioblk1 at virtio1
>>scsibus3 at vioblk1: 1 targets
>>sd1 at scsibus3 targ 0 lun 0: <VirtIO, Block Device, >
>>sd1: 664MB, 512 bytes/sector, 1360832 sectors
>>virtio1: msix shared
>>virtio2 at pci0 dev 6 function 0 "Qumranet Virtio Network" rev 0x00
>>vio0 at virtio2: address 00:a0:98:16:bb:2c
>>virtio2: msix shared
>>pcib0 at pci0 dev 31 function 0 "Intel 82371SB ISA" rev 0x00
>>isa0 at pcib0
>>isadma0 at isa0
>>com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo
>>pckbc0 at isa0 port 0x60/5 irq 1 irq 12
>>pckbd0 at pckbc0 (kbd slot)
>>wskbd0 at pckbd0 mux 1
>>pms0 at pckbc0 (aux slot)
>>wsmouse0 at pms0 mux 0
>>efifb at mainbus0 not configured
>>vscsi0 at root
>>scsibus4 at vscsi0: 256 targets
>>softraid0 at root
>>scsibus5 at softraid0: 256 targets
>>root on sd0a (2b7e2c838e3db383.a) swap on sd0b dump on sd0b
>>
>
>--
>Mark Patruck ( mark at wrapped.cx )
>GPG key 0xF2865E51 / 187F F6D3 EE04 1DCE 1C74 F644 0D3C F66F F286 5E51
>https://www.wrapped.cx
>
--
Mark Patruck ( mark at wrapped.cx )
GPG key 0xF2865E51 / 187F F6D3 EE04 1DCE 1C74 F644 0D3C F66F F286 5E51
https://www.wrapped.cx
No comments:
Post a Comment