Pf congestion troubleshooting

Hello,

We are experiencing congestion issues with PF and I would like some help finding the cause.
Here is what i have been able to gather so far:


ROOT:host:/root > pfctl -sm
states hard limit 600000
src-nodes hard limit 60000
frags hard limit 12000
tables hard limit 10000
table-entries hard limit 200000
pktdelay-pkts hard limit 10000
anchors hard limit 512

########################################################
ROOT:host:/root > pfctl -si
Status: Enabled for 1 days 11:41:03 Debug: err

Interface Stats for vlan0 IPv4 IPv6
Bytes In 2373225842545 464
Bytes Out 578501403973 0
Packets In
Passed 1993286988 0
Blocked 24490537 6
Packets Out
Passed 884448549 0
Blocked 50612 0

State Table Total Rate
current entries 145445
half-open tcp 9914
searches 14965499999 116496.6/s
inserts 145242314 1130.6/s
removals 145096869 1129.5/s
Counters
match 227954844 1774.5/s
bad-offset 0 0.0/s
fragment 183 0.0/s
short 30035 0.2/s
normalize 14897 0.1/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 11735216 91.4/s
ip-option 166 0.0/s
proto-cksum 0 0.0/s
state-mismatch 109522 0.9/s
state-insert 4 0.0/s
state-limit 16 0.0/s
src-limit 246 0.0/s
synproxy 0 0.0/s
translate 2838 0.0/s
no-route 0 0.0/s

#######################################################
ROOT:host:/root > vmstat -m | grep -E 'pf|Fail'
64 devbuf, pcb, rtable, pf, ifaddr, sysctl, counters, vnodes, UFS mount,
256 devbuf, rtable, pf, ifaddr, sysctl, counters, ioctlops, iov, vnodes,
1024 devbuf, pcb, pf, ifaddr, counters, ioctlops, iov, mount, shm, ACPI,
2048 devbuf, pcb, pf, ioctlops, iov, UFS mount, ACPI, file desc, VM swap,
4096 devbuf, pcb, pf, ifaddr, counters, ioctlops, iov, UFS mount,
16384 devbuf, pf, iov, dirhash, NFS daemon, MSDOSFS mount, ttys, temp
32768 devbuf, pf, UFS quota, UFS mount, ISOFS mount
pf 217 39K 71K629146K 10535077 0 64,256,1024,2048,4096,16384,32768
Name Size Requests Fail InUse Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
pfrule 1344 33736 0 10499 2104 1095 1009 1705 0 8 0
pfsrctr 152 3814 0 12 8 7 1 2 0 8 0
pfsnitem 16 17176 0 6 272 271 1 1 0 8 0
pfstate 344 145265949 0 159872 190473 175651 14822 20248 0 8 7
pfstkey 128 159453801 0 186551 23265 16870 6395 8271 0 8 2
pfstitem 24 159271270 0 186533 2146 915 1231 1501 0 8 0
pfruleitem 16 168209214 0 105937 700 229 471 579 0 8 0
pftag 88 44 0 44 1 0 1 1 0 8 0
pfanchor 1288 1589 0 1 34 33 1 10 0 8 0
pfrktable 1344 2597 0 692 163 3 160 163 0 8 0
pfrke_plain 168 19180 0 10818 834 340 494 834 0 8 0
pfosfpen 112 2142 0 714 21 0 21 21 0 8 0
pfosfp 40 2142 0 423 5 0 5 5 0 8 0
pffrent 40 2116813 0 0 279 278 1 3 0 8 1
pffrnode 88 906282 0 0 276 275 1 1 0 8 1
pffrag 232 1036002 0 0 422 421 1 13 0 482 1

#######################################################
ROOT:host:/root > netstat -i
Name Mtu Network Address Ipkts Ifail Opkts Ofail Colls
bnxt0 9000 <Link> bc:97:e1:d8:55:b0 1529467486 0 2492418876 40 0
bnxt1 9000 <Link> bc:97:e1:d8:55:b0 1311040429 0 2260699681 0 0
mcx0 9000 <Link> 04:3f:72:b8:bf:0a 1127074494 0 499148751 0 0
mcx1 9000 <Link> 04:3f:72:b8:bf:0a 1198061364 0 495767696 0 0
ixl0 9000 <Link> 40:a6:b7:3d:ac:60 1464092217 0 1262042851 0 0
ixl1 9000 <Link> 40:a6:b7:3d:ac:60 1716503824 0 1267250134 0 0
trunk0 9000 <Link> bc:97:e1:d8:55:b0 2840496912 0 4753118125 131422 0
trunk1 9000 <Link> 04:3f:72:b8:bf:0a 2325126977 0 994908032 4219 0
trunk2 9000 <Link> 40:a6:b7:3d:ac:60 3180587032 0 2529286504 98156 0
vlan0 1500 <Link> 04:3f:72:b8:bf:0a 2324523408 0 994911784 3752 0
vlan0 1500 10.90/16 10.90.0.10 2324523408 0 994911784 3752 0
vlan1 1500 <Link> 40:a6:b7:3d:ac:60 1725034503 0 1757650331 92484 0
vlan1 1500 10.1/16 10.1.0.250 1725034503 0 1757650331 92484 0
vlan10 1500 <Link> bc:97:e1:d8:55:b0 841039615 0 1905162366 31036 0


Thanks for your help.
Marc