Restrict SSH to local network only except for Git users?

I have a pretty nifty network setup that allows me to host from home via
WireGuard.
But there's one thing I'm struggling with.
Because for security reasons, I made it impossible for people outside
the network to connect via SSH, but for Git to function properly, I need
to allow SSH only for git@(DOMAIN) or git@(PUBLIC IP), and redirect that
to my home network so they can do stuff like "git pull", "git push", and
all the other fancy stuff.

My pf.conf rules look like this:
> pass in quick on wg0 proto tcp from 192.168.0.0/24 to any port 22
> pass in on $externalinterface proto tcp from any to $externalip port 22 rdr-to $internalip
> block in quick on egress proto tcp from any to any port 22

And my sshd_config:
> AllowUsers lain@192.168.0.0/24
> AllowUsers git@(DOMAIN)
> AllowUsers git@(PUBLIC IP)

Where exactly am I doing wrong here?