Understanding PF behavior

Hi guys,


So far I have spent a week on this and I feel like I'm not progressing,
now I just feel like I'm banging against a brick wall.


To start with, I managed to send icmp echos over my WAN link through
ISP-B within the same routing domain rdomain 2.


I then started looking at inter-Rdomain communication. The only
reference material I have got outside of the man pages are a few
outdated websites and other material which at least have gotten me in
the right direction.


From here on I have two issues.


The first issue is that my WAN communication for rdomain 2 broke. I
haven't adjusted any rules at all so the original working ruleset is
still in place.


Currently what I can see is tcpdump telling me that I have icmp
echo-reply packets on my test vlan for rdomain 2. I have checked pflog0
and the external IF for ISP-B using tcpdump and all seem normal too with
lots of "pass" statements and of course NAT is being hit.


Unfortunately the machine sending the icmp requests is claiming that
there is 100% packet loss.


I don't see any relevant "Block" information in tcpdump when initiating
"tcpdump -enipflog0 host x.x.x.2" - where x.x.x.2 is my test machine
address.


What could be  going on here? I have considered the fact that it maybe a
rule blocking on the outbound side of the interface (local test vlan),
but I tried adding a generic: pass out quick on $test_vlan rule which
didn't seem to do anything at all :-(



The second issue I have is to do with the routing domains. At the moment
I have a ruleset which allows me to get from my test vlan to one of the
internal vlans on rdomain0. I can verify with tcpdump that the pinged
machine can see the icmp packets and respond to them. What I am seeing
however in pflog0 is the lo0 is blocking traffic outbound.


As this is the reference guide I'm following:
https://www.packetmischief.ca/2011/09/20/virtualizing-the-openbsd-routing-table/


I'm using y.y.y.y/24 subnet for the icmp destination in rd0


I created routes to the loopback addresses:


route -T 0 add y.y.y/24 127.0.0.1

route -T 2 add y.y.y/24 127.0.0.1


I found that if I didn't add this rule, information to the source of the
icmp echos would be routed to my egress interface on ISP-A (existing ISP):

route -T 0 add x.x.x/24 127.0.0.1


Now, the unfortunate part is that I am seeing:

rule 2/(match) block out on lo0: y.y.y.N > x.x.x.2 icmp: echo reply


The thing I don't understand is that adding: pass out log quick on lo0

unfortunately doesn't seem to do anything??


Just to confirm that "set skip on lo" is disabled additionally.


If anyone could help in suggesting ideas of what to look at because I
have tried many things up till now and none have worked so I'm probably
approaching the problem in the wrong way.



Another quick question regarding the output of tcpdump. When it says
"rule 2" which rule is it referring to and how to tell this information?
In the past I recall the rule numbers given by tcpdump where the actual
rule line numbers as were written in pf.conf. Now I am completely
confused of what this means...

If I use "pfctl -sr" to show the rules, the second rule inline is
actually: "match in all scrub (no-df)" ???


Many thanks.


Kaya