After some more work this morning we have managed to extract the
information from tcpdump of the full LS-Update packet, we couldn't see it
on bsd, but running:
tcpdump -v -r ~/Downloads/ospf.pcap on osx did the trick.
What we are seeing is that a pair of firewalls are both sending updates
like this:
07:16:09.346525 IP (tos 0xc0, ttl 1, id 47473, offset 0, flags [+], proto
OSPF (89), length 1500)
x.x.x.x > ospf-dsig.mcast.net: OSPFv2, LS-Update, length 1480 [len 1672]
Router-ID x.x.x.x, Backbone Area, Authentication Type: simple (1)
Simple text password: dslkfjld, 1 LSA
LSA #1
Advertising Router x.x.x.x, seq 0x8000006e, age 0s, length 1624
Router LSA (1), LSA-ID: x.x.x.x
Options: [External]
Router LSA Options: [ASBR]
Stub Network: 10.128.32.128, Mask: 255.255.255.128
topology default (0), metric 10
Stub Network: 10.128.9.0, Mask: 255.255.255.128
*{ another 50 or so networks here}*
Each time we get one of these updates the DR logs the lsa_check: bad age.
Another 5 or so seconds later the same LS-Update comes in with the same seq
number. This appears to continue indefinitely. Our only fix appears to be
restarting ospfd on the routers.
Does anyone have an idea what is going wrong here?
Something we have considered being a problem is that we do have many
interfaces, we have 90 or so, so the LS-Update packets are quite large and
do get fragmented, as we are using a 1500mtu.
The fact that ospfd sees the age and complains though makes us think this
is not a problem.
Cheers
Richard
On Mon, May 4, 2020 at 6:12 PM Richard Chivers <r.chivers@zengenti.com>
wrote:
> Hi,
>
> Following on from the OSPF issue we were seeing in 5.8, we have built a
> vagrant lab with a complete replica of our production network in order to
> test config against 6.6 (latest syspatch applied) and test a number of
> scenarios.
>
> All in all everything has gone well, and other than some minor config
> enhancements, everything is fundamentally working.
>
> The original issue we had was routes not being advertised beyond the DR,
> when there were situations like a network blip or restart of the ospf
> process on another router/firewall.
>
> Since moving to 6.6 we have been able to recreate the same situation we
> have had in production, we do this by doing a "rcctl restart ospfd" on the
> DR, typically a few times.
>
> Eventually other routers start logging as follows:
>
> May 4 15:44:19 va-l1-tun ospfd[75371]: lsa_check: bad age
> May 4 15:44:19 va-l1-tun ospfd[75371]: lsa_check: bad age
> May 4 15:44:24 va-l1-br-02 ospfd[27625]: lsa_check: bad age
> May 4 15:44:24 va-l1-br-02 ospfd[27625]: lsa_check: bad age
> May 4 15:44:24 1 va-l1-tun ospfd[75371]: lsa_check: bad age
>
> If we run a tcpdump using tcpdump -i vio0 -s 1500 -w /tmp/ospf.pcap proto
> ospf, we can then see the ospf hello packets fully in wireshark, but the LS
> update packets are fragmented so we can not see the full detail or what is
> being passed from the relevant neighbor.
>
> We have tried to increase the verbosity of logging using "ospfctl log
> verbose", but still we are unsure which lsa update is incorrect.
>
> The only way we have found to stop these logs from appearing is to "rcctl
> restart ospfd" on various boxes until it stops.
>
> What we are hoping for help with is diagnosing exactly which record has
> the lsa_check: bad age, and understanding whether this should in effect
> clear itself for example.
>
> We have looked at the source code, but do not fully understand the flow
> beyond the check itself in lsa_check.
>
> We are wondering if there is something fundamentally wrong with our
> config, but it is pretty simple. Effectively a set of connected routers in
> a single area with one of the hops having a backup across the internet with
> a GRE tunnel. At most we are only ever 3 hops away between a source and
> destination.
>
> We have also on occasion seen "seq num mismatch, bad flags" messages, but
> these have appeared to clear themselves.
>
> Thanks
>
No comments:
Post a Comment