Sunday, December 05, 2021

Memory protection and the push instruction (amd64)

So this is a fairly esoteric question, and I expect the answer might
be just as esoteric.

I have a little toy fiber/stackless coroutine library that I made a
few years ago and have been using in some of my hobby projects.
(https://github.com/slembcke/Tina) The recent 7.0 release rekindled my
interest in OpenBSD, so I tried building my current game project for
it, and I was quite pleased that it mostly just needed a few ifdefs
changed. Well, except for the fiber library... There were two issues,
the first was trivial to fix. I just had to switch to mmap() +
MAP_STACK to allocate the stack buffers since OpenBSD requires it. The
second I found a workaround for easily enough, but I'm really curious
why it happens. If I mmap() a buffer, I can mark the final page as
PROT_NONE as a guard page. Then if I set %rsp to the guard page's
address and push a value, it subtracts and then writes the value just
below the guard page as expected on Linux/FreeBSD/Mac. However, on
OpenBSD this fails with a segfault. Either starting 16 bytes lower, or
replacing the push instruction with a sub and a mov to emulate it
works fine. So there is an easy workaround, but my question is why? I
assumed pagefaults were implemented in hardware by the MMU as an
interrupt or something during reads and writes. Is this a subtle bug
in OpenBSD, and if it is, how does it even know since the write
doesn't even affect the protected page?! (Protected memory is black
magic already! Hah!) I would assume that whatever is happening here
would have to be handled when initializing the stack for processes or
threads too.

- Scott

No comments:

Post a Comment