Re: pf queues

On Wed, 29 Nov 2023 00:12:02 +0300
4 <babut@yandex.ru> wrote:

> i haven't used queues for a long time, but now there is a need.
> previously, queues had not only a hierarchy, but also a priority. now
> there is no priority, only the hierarchy exists.

It took me quite some time to wrap my head around this, having been
accustomed to HFSC up until 5.5. One can probably find a lot of my
emails in misc@ archives from that time.

Nowadays I am matching traffic to prio and queue by protocol and/or
destination port only. Anything not explicitly matched goes to lowest prio
queue and logged even when passed, so I can inspect if there are any
types of traffic which should be put into appropriate prio / queue. All
the ACKs except those in lowest prio queue get highest (7) priority,
stuff in lowest prio have lowest prio for ACKs as well.

# QUEUE MATCHES
match proto icmp set prio ( 6 7 ) queue ( 6-fly 7-ack ) tag queued
match proto ospf set prio ( 6 7 ) queue ( 6-fly 7-ack ) tag queued
match proto tcp to port $q_fly set prio ( 6 7 ) queue ( 6-fly 7-ack ) tag queued
match proto udp to port $q_fly set prio ( 6 7 ) queue ( 6-fly 7-ack ) tag queued
match proto tcp to port $q_sprint set prio ( 5 7 ) queue ( 5-sprint 7-ack ) tag queued
match proto udp to port $q_sprint set prio ( 5 7 ) queue ( 5-sprint 7-ack ) tag queued
match proto tcp to port $q_run set prio ( 4 7 ) queue ( 4-run 7-ack ) tag queued
match proto udp to port $q_run set prio ( 4 7 ) queue ( 4-run 7-ack ) tag queued
match proto tcp to port $q_walk set prio ( 2 7 ) queue ( 2-walk 7-ack ) tag queued
match proto udp to port $q_walk set prio ( 2 7 ) queue ( 2-walk 7-ack ) tag queued
match proto tcp to port $q_crawl set prio ( 1 7 ) queue ( 1-crawl 7-ack ) tag queued
match proto udp to port $q_crawl set prio ( 1 7 ) queue ( 1-crawl 7-ack ) tag queued
match log ! tagged queued set prio 0 queue ( 0-other )

I used to hear a lot of "Don't queue inbound (to the interface)". I
heard a few "you can't queue it inbound (to the interface) - you
already received it. But response to that inbound request will be
assigned to appropriate queue." In my observation the latter appears to
be correct approach. Hence, no 'match out proto...' but 'match
proto...'.

I used to hear a lot of "Don't queue (traffic received on Internet
interface) outbound on LAN interface. You already received packets,
why drop it?". But this is necessary if one wants to give full
bandwidth to e.g. p2p when no other traffic is on the wire, but
throttle it in lieu of DNS or web or IM traffic, once it kicks in.

So here's how my queues look like for a home gateway with 300/30Mbit
link:

# QUEUES
queue wan on $if_ext bandwidth 24M min 24M max 24M
queue 7-ack parent wan bandwidth 6M min 1500K max 18M qlimit 512
queue 6-fly parent wan bandwidth 2M min 500K max 18M qlimit 512
queue 5-sprint parent wan bandwidth 2M min 500K max 18M qlimit 512
queue 4-run parent wan bandwidth 2M min 500K max 18M qlimit 512
queue 2-walk parent wan bandwidth 2M min 500K max 18M qlimit 2048
queue 1-crawl parent wan bandwidth 1M min 250K max 18M qlimit 2048
queue 0-other parent wan bandwidth 1M min 250K max 18M qlimit 4096 default

queue lan on $if_int bandwidth 240M min 240M max 240M
queue 7-ack parent lan bandwidth 60M min 15M max 180M qlimit 512
queue 6-fly parent lan bandwidth 20M min 5M max 180M qlimit 512
queue 5-sprint parent lan bandwidth 20M min 5M max 180M qlimit 512
queue 4-run parent lan bandwidth 20M min 5M max 180M qlimit 512
queue 2-walk parent lan bandwidth 20M min 5M max 180M qlimit 2048
queue 1-crawl parent lan bandwidth 10M min 2500K max 180M qlimit 2048
queue 0-other parent lan bandwidth 10M min 2500K max 180M qlimit 4096 default

Make sure to measure your real download and upload bandwidths, allocate
no more than 90% of it (at worst of its times if it varies).

I get satisfactory results when:
- I set bandwidth, min and max to the same value on parent queue
- Sum of child queue bandwidths does not exceed parent queue bandwidth
- I allocate child queue min values to 1/4 of child queue bandwidth
values
- I allocate child queue max values to (parent queue - sum of child
queue min values) or less
- I raise qlimit somewhat for high priority queues, and quite a bit for
low priority queues
- I flush all the states with 'pfctl -f states' after I do changes to
ruleset at initial stage of early testing
- I keep an eye on 'systat queues / states / rules' to understand
exactly which rule triggers assignment to which queue.

Now all of the above is fine for home gateway with just "internet" and
"lan". Things get much more complicated if there are multiple VLANs on
internal interface, GRE / GIF of wireguard tunnels on external
interfaces etc.

I once had the privilege to sit with Henning, author of 'pf megapatch'
who introduced new queuing mechanism. I complained new stuff is not
well documented, and asked if he could explain it better to me. He said
something along the lines "I have no idea. It works for me. All I
know is in the manpage and the code is available in CVS. Try if it works
for you. If it doesn't and you know what should be improved send a
patch". Upon hearing this, I was enlightened :)

I hope above will be helpful.

Best regards,

--
Before enlightenment - chop wood, draw water.
After enlightenment - chop wood, draw water.

Marko Cupać
https://www.mimar.rs/