Re: pf, relayd, TCP keep alive and NAT, oh my!

On Tue, Jun 01, 2021 at 10:25:38AM +1000, Cameron Simpson wrote:
> Can I enforce or implement TCP keep alives on a TCP stream via my
> firewall?
>
> Background:
>
> I've got a client with an OpenBSD firewall and a Telstra NBN modem as
> their modem.
>
> Their IMAP server is upstream in the cloud (Unbuntu, courier imap). I
> have this odd problem which I am beginning to suspect is the NBN modem
> getting bored and dropping its NAT entries. Let me explain...
>
> At the firewall end I see about 30 ESTABLISHED connections to the IMAP
> server. At the IMAP server I see over 500, which is about where the IMAP
> service stops accepting new connections, leading to errors from the
> client mail readers.
>
> My current theory is that the IMAP client connections issue the IMAP
> IDLE command and go passive, waiting for email notifications from the
> server. So we have an idle TCP connection across the firewall and
> across the NBN modem (which NATs).
>
> My conjecture is that at some point the modem discards idle connection
> states. (This could just as well happen at any other intermediate
> stateful router too.) After that event, the client end does something
> which tries to use the connection, gets an RST from the modem, clean
> tidyup happens on the client and in the firewall.
>
> At the server end, none of this is seen and the imapd just sits around
> idle, never releasing the connection and never stopping the matching
> daemon process. This gradually rises to hit the server's configured
> connection limit and it stops accepting new things.
>
> If I had TCP keep alive turned on, both ends might tidy themselves up.
> I can't enable that on the clients (various mail readers) or,
> apparently, on the server configuration. I can't do it in PF because PF
> just copies packets. I can't seem to do it in relayd either, though that
> seems the obvious way to intercept the connection for this purpose.
>
> Any suggestions?

Make sure you use 'block return' at least for the imap connections. This
way when the state is dropped the firewall will issue a RST packet to the
server which will close the connection.

On OpenBSD there is the 'net.inet.tcp.always_keepalive' sysctl to enable
keepalive by default. So that is something you can enable on the IMAP
server to force keep-alive on there. Other systems have similar knobs.

--
:wq Claudio