rdomains finally working!!

Hey guys,


I can't spend too much time right now in writing up what I have done
right now as I'm just about to head out of town to a local nature
reserve in order to clear my head, but basically things are working!!


At this stage with such limited documentation on the topic I wonder if I
should do some kind of a writeup of it to share my own experiences?
Hopefully later in the week and I'll pop it up on my GDrive.... maybe it
can be turned into part of the doc or faq or an example page on the OBSD
website?


What I can gloss over right now is that the setup is very sensitive with
minute changes having drastic effects. To be honest I don't even know
how or when things started working but I checked my laptop test machine
and icmp packet responses were being received properly.


Running tcpdump on pflog does lie sometimes. It will often suggest
"block" when in fact the problem is routing. Another issue that I came
across is that the loopback addresses don't inter-route. I don't know
how to make that one work at all?


Taking an excerpt from the website I was following:

https://www.packetmischief.ca/2011/09/20/virtualizing-the-openbsd-routing-table/

Citing:

Creating a loopback interface in rdomain 2 so that Host 1 can talk to
Host 2 would look like:

ifconfig lo2 rdomain 2 127.0.0.1
route -T 2 add 192.168.1/24 127.0.0.1
Since lo2 is created inside rdomain 2, the IP address assigned to it
doesn't conflict with lo0 in rdomain 0.


Sure I can see traffic inside one of the loopbacks and tcpdump does
claim "pass out" but then nothing else happens. The other loopback
interfaces have no traffic in them and the destination network has no
traffic either.


Over the weekend having gotten completely fedup I decided to try to use
one of the rdomains as a transit system and connected my Cisco 2801
BGP/IPSec/GRE test bed to the vlan in question and attempted to use it
as a simple WAN Edge gateway. At first I managed to ping the interface
of the Cisco, as I set OSPF up so that I didn't need to bother with
static routes. As soon as I moved the rdomain of the vlan from 0 over to
3, everything stopped working and I couldn't find a way to use OSPF
within two separate routing tables. The man page does have a setting for
this in the config but there is no mention if you can use 2x instances
between rdomains.....

After that I fought with static routes and of course the strange
behavior kicked in where sending icmp packets between the Cisco and
OpenBSD machines just wasn't working. I really didn't understand this at
all as they are Next-Hop connected devices. No specific PF rule and a
default route was set to be the interface of the Cisco routers internal
IP address.


I ended up moving the setup back to what I had originally with 3x
rdomains and again I could not communicate between rdomain 2 and rdomain
3, though I did have communication to rdomain 0 from rdomain 2. Having
done some more thinking about it and looking at the Cisco setup I
decided to try creating an extra vlan in rdomain 3 with the intention of
using it as a transit vlan. I already had success in pinging the
internal IP address of the OpenBSD machine from domain 2 to domain 3, so
I figured what the heck I'll give it a try.

It did work but I still couldn't ping the public address.


I then moved the test machine from rdomain 2 to rdomain 3 to see if that
would help with internal communication between the machine and the
public IP address. Oddly enough it did work and local traffic sent to
rdomain 0 was still working too...... though I was still unable to get
any response from public address.


This is a bit of a black hole for me because I left the system for a
while and came back to find that it actually started to work?? It isn't
logical of course but I can't recall if I made changes in the meantime
and simply wasn't aware of anything happening.


After testing a bit and confirming that in fact things were working and
I was *not* sending traffic over my existing PlusNET WAN link, I
proceeded to migrate my OpenVPN tunnel across. This proved to be tricky
as I kept running into dreaded "tcpdump -enipflog" "block in on tun0"
issues.... in fact that turned out to be a routing problem instead of a
PF issue.

Currently this is working too but with some strange adjustments to PF.
The current 'pass' per port rulesets don't seem to work even with the
'rtable (x)' clause added. What does seem to work however is a generic
'pass to <ip-range> rtable (x)' ruleset


I'll probably need to study my rules further in order to try to
understand the behavior here....


Anyway, am gona shoot over to the Warburg Nature Reserve now and
continue with my photography to hopefully share more images with the
BBOWT later. Oh and of course to try to get rid of my headache too ;-)


Thanks to everyone for trying to help on a subject that contains very
limited documentation and examples. Like I said hopefully if I can do
some kind of write up later this week perhaps it could be integrated
into the OBSD website or something??


Regards,


Kaya