Friday, December 01, 2023

relayd checks and uses disabled hosts

Hi,

I have a strange behavior on my relayd servers. Relayd continues checking disabled hosts. I see it on backend server's logs.
If relayd detects a down -> up of the service it re-adds the hosts in the table and passes traffic to the disabled hosts.

Status remains disabled.

Setup is with redirects.

table <ldap> { ldap1 retry 2, ldap2 retry 2 }
redirect ldap {
   listen on $ldap_addr port ldaps
   pftag RELAYD_ldap
   forward to <ldap> port 1636 mode least-states check icmp check script "/usr/local/sbin/check_ldap_c" demote 0relay timeout 2000
   session timeout 432600
}

On load balancer hosts I see:
pfctl -a 'relayd/ldap' -t ldap -Tshow
   ldap1_IP
   ldap2_IP

If I do
relayctl host dis ldap2

I see in logs

Dec  1 13:11:24 relayd[59724]: table ldap: 0 added, 1 deleted, 0 changed, 0 killed

# relayctl show sum|grep ldap
1       redirect        ldap                                    active
1       table           ldap:1636                               active (1 hosts)
1       host            ldap1                           100.00% up
2       host            ldap2                                   disabled

# pfctl -a 'relayd/ldap' -t ldap -Tshow
   ldap1_IP (only)

So far, so good. However...

However, when I actually close the service on server ldap2 I see:

Dec  1 13:12:27 relayd[42873]: host ldap2, check script (766ms,script failed), state up -> down, availability 98.29%
Dec  1 13:12:27 relayd[71859]: table ldap: 0 added, 0 deleted, 0 changed, 0 killed

Now, when I restart the server or the service on ldap2:

Dec  1 13:17:08 relayd[42873]: host ldap2, check script (987ms,script ok), state down -> up, availability 98.28%
Dec  1 13:17:12 relayd[71859]: table ldap: 1 added, 0 deleted, 0 changed, 0 killed

# relayctl show sum|grep ldap2
2       host            ldap2                                   disabled

Hosts is shown as disabled, but it's added the table.

# pfctl -a 'relayd/ldap' -t ldap -Tshow
   ldap1_IP
   ldap2_IP

again:

# relayctl host dis ldap2
command succeeded.

# pfctl -a 'relayd/ldap' -t ldap -Tshow
   ldap1_IP
   ldap2_IP

During this whole time while ldap2 is disabled I keep seeing in ldap2's logs connects from the load balancer although it's disabled. Logs from the check script.

When the check sees the service down->up, it re-enables the host although in summary it's still stated as disabled.
Clients are also coming now apart from the check script.

If I re-enable the disabled host:

# relayctl host en ldap2
command succeeded

Dec  1 13:24:35 relayd[99810]: host ldap2, check script (796ms,script ok), state unknown -> up, availability 100.00%
Dec  1 13:24:39 relayd[59724]: table ldap: 0 added, 0 deleted, 0 changed, 0 killed

I checked web csv but can't see any related change on relayd...

On August and 7.3 this didn't happen.

Giannis

No comments:

Post a Comment