seeing carp interface state change for unknown reason ; cluestick hunting

Hi all -

I'm trying to setup a pair of ERL3 octeon routers in master/standby
mode via carp/pfsync to route traffic from my internal lan to the
internet. I've seen strange behavior wrt carp on these machines, so
in an attempt to reduce the problem, I've removed one completely.

Even with only a single box (ERL3-01) on the network configured as a
carp member, the carp interface state periodically changes (as seen
from ifstated(8)).

I'm wondering if disconnecting the other ERL3 device is a valid isolated test.
1. Will/might this cause issues with the carp device, as it cannot
determine state from any other host?
2. Will/might this cause issues as it cannot send/receive pfsync
updates (the other node is disconnected).
3. Is there something else in my setup causing carp to fail here?
4. Could this be hardware/temperature related to this ERL3? Wouldn't
I see an additional error in dmesg if the physical device (cnmac2)
failed periodically?

I'd appreciate any pointers here...I feel like I'm missing something dumb.

Thanks in advance.

Bryan

Here are some of my configs. If I've missed including something
critical to help describe my setup, please let me know and I'll add
it.

## Help me OBSD-Misc Kenobi. You're my only hope. ##

erl3-01# uname -a
OpenBSD erl3-01.siliconvortex.com 6.8 GENERIC#522 octeon

erl3-01# dmesg
...
carp1: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER

erl3-01# tail mbox
Mon, 1 Feb 2021 06:49:26 +0000 (UTC)
From: Charlie Root <root@erl3-01.siliconvortex.com>
Date: Mon, 1 Feb 2021 06:49:25 +0000 (UTC)
To: root@localhost
Subject: carp master changed
Message-ID: <515eb74cff427290@erl3-01.siliconvortex.com>
Status: RO

master is now erl3-01.siliconvortex.com


erl3-01# sysctl -a | grep carp
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=2

erl3-01# cat /etc/hostname.carp1
#carp for lan side
192.168.122.1/23 carpdev vlan100 vhid 1 pass somethinglongandsecret

erl3-01# cat /etc/hostname.vlan100
vnetid 100 parent cnmac2
up

erl3-01# cat /etc/hostname.cnmac2
inet 192.168.1.253 255.255.254.0

erl3-01# cat /etc/hostname.pfsync0
up syncdev cnmac1

erl3-01# cat /etc/hostname.cnmac1
inet 10.10.200.1 255.255.255.252

erl3-01# cat /etc/ifstated.conf
# Initial State
init-state auto

# Macros
if_carp_up="carp1.link.up"
if_carp_down="!carp1.link.up"

state auto {
if $if_carp_up {
set-state master
}

if $if_carp_down {
set-state backup
}
}

state master {
init {
run "echo master is now `hostname` | mail -s 'carp master changed'
root@localhost"
}

if $if_carp_down {
set-state backup
}
}

state backup {
init {
run "echo backup is now `hostname` | mail -s 'carp master changed
root@localhost"
}

if $if_carp_up {
set-state master
}
}

erl3-01# cat /etc/pf.conf
# adopted from https://www.openbsd.org/faq/pf/example1.html
wan_dev = cnmac0
lan_dev = cnmac2
carp_dev = vlan100
pfsync_dev = cnmac1
table <martians> { 0.0.0.0/8 10.0.0.0/8 127.0.0.0/8 169.254.0.0/16 \
172.16.0.0/12 192.0.0.0/24 192.0.2.0/24 224.0.0.0/3 \
192.168.0.0/16 198.18.0.0/15 198.51.100.0/24 \
203.0.113.0/24 }

# carp
pass quick on $lan_dev proto carp keep state (no-sync)

# pfsync
pass quick on $pfsync_dev proto pfsync keep state (no-sync)

set block-policy drop
set loginterface $wan_dev
set skip on lo0

match in all scrub (no-df random-id max-mss 1440)

# redirect DNS queries to localhost
pass in quick on { $carp_dev $lan_dev } proto { udp tcp } from any to
any port domain rdr-to 192.168.1.253 port domain

# NAT to the world
match out on $wan_dev inet from !($wan_dev:network) to any nat-to ($wan_dev:0)

antispoof quick for { $wan_dev }

# martians
block in quick on $wan_dev from <martians> to any
block return out quick on $wan_dev from any to <martians>

block all

# manage buffer bloat
queue outq on $wan_dev flows 1024 bandwidth 3M max 3M qlimit 1024 default
queue inq on $lan_dev flows 1024 bandwidth 45M max 45M qlimit 1024 default

pass out quick inet

pass in on { $carp_dev $lan_dev } inet