At a guess, route-to is confused by the same ip, but I haven't looked at the internals.
Maybe try adding pair interfaces (with different addresses) to each rdomain,
and you can use route-to to select between them.
You already have default route set in each rdomain, so it will find its way from there.
eg.
# /etc/hostname.pair1
group pinternal
rdomain 1
inet 10.255.1.1 255.255.255.0
!/sbin/route -T1 add <your internal subnet(s)> 10.255.1.2
# /etc/hostname.pair11
group pinternal
inet 10.255.1.2 255.255.255.0
patch pair1
# /etc/hostname.pair2
group pinternal
rdomain 2
inet 10.255.2.1 255.255.255.0
!/sbin/route -T2 add <your internal subnet(s)> 10.255.2.2
# /etc/hostname.pair12
group pinternal
inet 10.255.2.2 255.255.255.0
patch pair2
# /etc/pf.conf
...
pass on pinternal
...
pass in quick on { $if_int } to any route-to { 10.255.1.1, 10.255.2.1 } \
round-robin set prio (3,6)
Have not tested exactly this, but similar to my current setup.
Might not need the static routes, if the right pf magic is happening.
-Phil
On 28/11/18 8:18 am, Andrew Lemin wrote:
> Hi,
>
> So using the information Stuart and Andreas provided, I have been testing
> this (load balancing across multiple VPN servers to improve bandwidth).
> And I have multiple VPNs working properly within there own rdomains.
>
> * However 'route-to' is not load balancing with rdomains :(
>
> I have not been able to use the more simple solution you highlighted Stuart
> (using basic multipath routing), as the tunnel subnets overlap.
> So I think this is a potential bug, but I need your wisdom to verify my
> working first :)
>
> Re; Load Balancing SSL VPNs using OpenBSD 6.4, with VPN TunX interfaces in
> unique rdomains (overlapping tunnel subnets)
>
> Configure sysctl's
> # Ensure '/etc/sysctl.conf' contains;
> net.inet.ip.forwarding=1 # Permit forwarding (routing) of packets
> net.inet.ip.multipath=1 # 1=Enable IP multipath routing
>
> # Active sysctl's now without reboot
> sysctl net.inet.ip.forwarding=1
> sysctl net.inet.ip.multipath=1
>
> Pre-create tunX interfaces (in their respective rdomains)
> # Ensure '/etc/hostname.tun1' contains;
> up
> rdomain 1
>
> # Ensure '/etc/hostname.tun2' contains;
> up
> rdomain 2
>
> # Bring up the new tunX interfaces
> sh /etc/netstart
>
> fw1# ifconfig
> tun1
>
> tun1: flags=8011<UP,POINTOPOINT,MULTICAST> rdomain 1 mtu 1500
> index 8 priority 0 llprio 3
> groups: tun
> status: down
> fw1# ifconfig tun2
> tun2: flags=8011<UP,POINTOPOINT,MULTICAST> rdomain 2 mtu 1500
> index 9 priority 0 llprio 3
> groups: tun
> status: down
>
> # Start all SSL VPN tunnels (in unique VRF/rdomain's)
> /usr/local/sbin/openvpn --config ./ch70.nordvpn.com.udp.ovpn --writepid
> /var/run/openvpn.tun1.pid --dev tun1 &
> /usr/local/sbin/openvpn --config ./ch71.nordvpn.com.udp.ovpn --writepid
> /var/run/openvpn.tun2.pid --dev tun2 &
> ('auth-user-pass' updated in config files)
>
> Each openvpn tunnel should start using 'rtable 0' for the VPN's outer
> connection itself, but with each virtual tunnel TunX interface being placed
> into a unique routing domain.
>
> This results in the following tunX interface and rtable updates;
> fw1# ifconfig
> tun1
>
> tun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> rdomain 1 mtu 1500
> index 6 priority 0 llprio 3
> groups: tun
> status: active
> inet 10.8.8.128 --> 10.8.8.1 netmask 0xffffff00
> fw1# ifconfig tun2
> tun2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> rdomain 2 mtu 1500
> index 7 priority 0 llprio 3
> groups: tun
> status: active
> inet 10.8.8.129 --> 10.8.8.1 netmask 0xffffff00
> fw1# route -T 1 show
> Routing tables
> Internet:
> Destination Gateway Flags Refs Use Mtu Prio
> Iface
> 10.8.8.1 10.8.8.128 UH 0 0 - 8
> tun1
> 10.8.8.128 10.8.8.128 UHl 0 0 - 1
> tun1
> localhost localhost UHl 0 0 32768 1
> lo1
> fw1# route -T 2 show
> Routing tables
> Internet:
> Destination Gateway Flags Refs Use Mtu Prio
> Iface
> 10.8.8.1 10.8.8.129 UH 0 0 - 8
> tun2
> 10.8.8.129 10.8.8.129 UHl 0 0 - 1
> tun2
> localhost localhost UHl 0 0 32768 1
> lo2
>
> # Test each tunnel - Ping the remote connected vpn peer within each rdomain
> ping -V 1 10.8.8.1
> ping -V 2 10.8.8.1
>
> Shows both VPN tunnels are working independently with the overlapping
> addressing :)
>
> # To be able to test each tunnel beyond the peer IP, add some default
> routes to the rdomains;
> route -T 1 -n add default 10.8.8.1
> route -T 2 -n add default 10.8.8.1
>
> # Test each tunnel - Ping beyond the connected peer
> ping -V 1 8.8.8.8
> ping -V 2 8.8.8.8
>
> Shows both VPN tunnels are definitely working independently with the
> overlapping addressing :)
>
> # Reverse routing - I have read in various places that PF's 'route-to' can
> be used for jumping rdomains's in the forward path of the session, but the
> reply packets need any matching route in the remote rdomain for the reply
> destination (the matching route is to ensure in the reply packet is passed
> through the routing table and gets into the PF processing, where PF can
> manage the return back to the default rdomain etc.
>
> But as I am using outbound NATing on the tunX interfaces, there is always a
> matching route for the reply traffic. And so a route for the internal
> subnet is not needed within rdomain 1 and 2.
>
>
> # Finally ensure '/etc/pf.conf' contains something like;
> if_ext = "em0"
> if_int = "em1"
>
> #CDR = 80 Down/20 Up
> queue out_ext on $if_ext flows 1024 bandwidth 18M max 19M qlimit 1024
> default
> queue out_tun1 on tun1 flows 1024 bandwidth 17M max 18M qlimit 1024 default
> queue out_tun2 on tun2 flows 1024 bandwidth 17M max 18M qlimit 1024 default
> queue out_int on $if_srx flows 1024 bandwidth 74M max 78M qlimit 1024
> default
>
> #MTU = 1500
> match proto tcp all scrub (no-df max-mss 1460) set prio (2,5)
> match proto udp all scrub (no-df max-mss 1472) set prio (2,5)
> match proto icmp all scrub (no-df max-mss 1472) set prio 7
>
> #NAT all outbound traffic
> match out on $if_ext from any to any nat-to ($if_ext)
> match out on tun1 from any to any nat-to (tun1) rtable 1
> match out on tun2 from any to any nat-to (tun2) rtable 2
>
> #Allow outbound traffic on egress for vpn tunnel setup etc
> pass out quick on { $if_ext } from self to any set prio (3,6)
>
> #Load balance outbound traffic from internal network across tun1 and tun2 -
> THIS IS NOT WORKING - IT ONLY USES FIRST TUNNEL
> pass in quick on { $if_int } to any route-to { (tun1 10.8.8.1), (tun2
> 10.8.8.1) } round-robin set prio (3,6)
>
> #Allow outbound traffic over vpn tunnels
> pass out quick on tun1 to any set prio (3,6)
> pass out quick on tun2 to any set prio (3,6)
>
>
> # Verify which tunnels are being used
> systat ifstat
>
> *This command shows that all the traffic is only flowing over the first
> tun1 interface, and the second tun2 is never ever used.*
>
>
> # NB; I have tried with and without 'set state-policy if-bound'.
>
> I have tried all the load balancing policies; round-robin, random,
> least-states and source-hash
>
> If I change the 'route-to' pool to "{ (tun2 10.8.8.1), (tun1 10.8.8.1) }",
> then only tun2 is used instead.. :(
>
> So 'route-to' seems to only use the first tunnel in the pool.
>
> Any advice on what is going wrong here. I am wondering if I am falling
> victim to some processing-order issue with PF, or if this is a real bug?
>
> Thanks, Andy.
>
>
>
No comments:
Post a Comment