> Why have you chosen to hide information that may be useful in debugging your problem?
I'm truly sorry for the inconvenience but I do have some concerns of security and privacy. I confirm it is not a broadcast address because it is the public IP of the server and this issue has a probability of 1% to happen. The address cannot just be a broadcast address at 1% of the time while not at the rest of 99%. I also double checked it by SSHing to the address I copied from the kdump, if it makes sense.
> So, since the manpage mentions blocking pf, I suggest the hypothesis "it returns EACCES because pf is blocking your packets". I can think of several ways to test that; what testing have you performed to confirm or rule out that possibility? "doas pfctl -d; run test; doas pfctl -e"?
This issue is really hard to reproduce because the application works at most of the time, but I think you are right. I'll be watching the pf log in next weeks.
> Alternatively: what's different about *that* call? Does every sento() call on that socket fail? What is special about that socket? If other sendto() calls succeed, what is different about that call? Earlier setsockopt() calls?
The call failed once after a few days of running. I have no idea of how to present this but it seems the second and third arguments of sendto call are always changing?
3058 myapp CALL sendto(5,0x1689f5f6100,0x5d,0x400<MSG_NOSIGNAL>,0x7f7fffff1144,0x10)
3058 myapp STRU struct sockaddr { AF_INET, xxx.xx.xxx.xx:yyyyy }
3058 myapp GIO fd 5 wrote 93 bytes
"\M^Ai9\M^N\M^U\M-@9 \M-7\M-B\M-m\M^C.\M-ut\M^E~)\M^H'w\M-A\M-p|\M-+DK\M-V\M-8v\M-~c\240\M-=y\M-E\M-*\M-+.\M-I\M-m\M^[\M-s\M-&\\\M^GwP\M-I\r\M^[\M-C\^]B\M-%\M-<\M-q\
\M-rk\M-_\M-g\M-9Y\^B\M-+Y\M-:K\M^X\^E/*\M-4_\M-x\at\M-{\M-$\M-[\^NU\\\M^HPk\M-i\\\M-M\M-H8t
"
3058 myapp RET sendto 93/0x5d
3058 myapp CALL kevent(3,0,0,0x168a7f0c000,128,0)
3058 myapp STRU struct kevent { ident=4, filter=EVFILT_READ, flags=0x61<EV_ADD|EV_CLEAR|EV_RECEIPT>, fflags=0<>, data=73, udata=0x2 }
3058 myapp RET kevent 1
3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0)
3058 myapp GIO fd 4 read 73 bytes
"\0\0\0\^BE\0\0E\M^_?\0\0@\^Q\M-9\M-:\M-@\M-(_\^D\^A\^A\^A\^A\M-x\M-U\0005\0001\M^U\M-Q\M-B\M-U\^A\0\0\^A\0\0\0\0\0\^A\^Bbt\^Ehenbt\^Ccom\0\0\^A\0\^A\0\0)\^D\M-P\0\
\0\M^@\0\0\0"
3058 myapp RET read 73/0x49
3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0)
3058 myapp GIO fd 4 read 79 bytes
"\0\0\0\^BE\0\0K\M-i\M^E\0\0@\^Qpo\M-@\M-(_\^D\^A\0\0\^A\M-Sy\0005\0007\M-8 \M-dB\^A\0\0\^A\0\0\0\0\0\^A\atracker\^Fcity9x\^Ccom\0\0\^A\0\^A\0\0)\^D\M-P\0\0\M^@\0\0\
\0"
3058 myapp RET read 79/0x4f
3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0)
3058 myapp RET read -1 errno 35 Resource temporarily unavailable
3058 myapp CALL read(7,0x7f7fffff07e0,0xc)
3058 myapp GIO fd 7 read 12 bytes
"4z.\M-"\M-)-\M-o\^E\M-v\^Q\M-5u"
3058 myapp RET read 12/0xc
3058 myapp CALL sendto(5,0x1690776aa00,0x61,0x400<MSG_NOSIGNAL>,0x7f7fffff1144,0x10)
3058 myapp STRU struct sockaddr { AF_INET, xxx.xx.xxx.xx:yyyyy }
3058 myapp GIO fd 5 wrote 97 bytes
"\M-)\M-,\M-r\M-b\^R\M-Jb\M-@\M-b\M-{\M^G"\M-Z\v\M-v\M-L\^^\M-BI\M-X\M-S\M-L]\M-&\^R%\M-?\M-\\M-kN\M^G:\M-NVI\M-@s\0#\M-V\M-}\M-h\M-T\M^C\^R\M-W\M-"\^?\M^Q[:u\M-=\
\M-Q%\M-c\M-+\M-xZ\^B\M-$\M-N\M-1\^Zy\M^D 9#\^Q\M--\M^Jb\M^T( \M^@\M^L;~\M-<\M-}0\M-G\M-`4z.\M-"\M-)-\M-o\^E\M-v\^Q\M-5u"
3058 myapp RET sendto 97/0x61
3058 myapp CALL read(7,0x7f7fffff07e0,0xc)
3058 myapp GIO fd 7 read 12 bytes
"\M-j\M->\M-9\^V\^U\^^\M-c/h',\M^D"
3058 myapp RET read 12/0xc
3058 myapp CALL sendto(5,0x1690776a600,0x67,0x400<MSG_NOSIGNAL>,0x7f7fffff1144,0x10)
3058 myapp STRU struct sockaddr { AF_INET, xxx.xx.xxx.xx:yyyyy }
3058 myapp GIO fd 5 wrote 103 bytes
"\^W\^\\M-B\M^E\M-~\M-c\M-3\M-f\M-X\M^Y\M-Z\^V\M-G\M-p\M^L\M^G\M-b\M-T\M^I\M-SWu\M-]\M^Jq\M^Q\M-Z\^Z\M^[\bW>`/fW\b\M^K\M-*\^]:E\M-T\M-[\M^X\M-5M\M^J\M-j\M-v\^^Z@:\
\M-QfQ\a-\M-\)\M-O$[\M-x]\b\M-Wh5\M^F\M^B|I> \M^CY\M-%\M-6'\M^?g9F?y\^T\M-}\^E\M-g\M-j\M->\M-9\^V\^U\^^\M-c/h',\M^D"
3058 myapp RET sendto 103/0x67
3058 myapp CALL kevent(3,0,0,0x168a7f0c000,128,0)
3058 myapp STRU struct kevent { ident=4, filter=EVFILT_READ, flags=0x61<EV_ADD|EV_CLEAR|EV_RECEIPT>, fflags=0<>, data=75, udata=0x2 }
3058 myapp RET kevent 1
3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0)
3058 myapp GIO fd 4 read 75 bytes
"\0\0\0\^BE\0\0GS\M^L\0\0@\^Q\^Fm\M-@\M-(_\^D\^A\0\0\^A\M-n\M-)\0005\0003m\M-_N\M-#\^A\0\0\^A\0\0\0\0\0\^A\^Cbtx\aanifilm\^Btv\0\0\^A\0\^A\0\0)\^D\M-P\0\0\M^@\0\0\0"
3058 myapp RET read 75/0x4b
3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0)
3058 myapp RET read -1 errno 35 Resource temporarily unavailable
3058 myapp CALL read(7,0x7f7fffff07e0,0xc)
3058 myapp GIO fd 7 read 12 bytes
"\M^T\M-|\M-e\M-Fk-\M^P\M-u}\M-j\M^U="
3058 myapp RET read 12/0xc
3058 myapp CALL sendto(5,0x1690776a500,0x63,0x400<MSG_NOSIGNAL>,0x7f7fffff1144,0x10)
3058 myapp STRU struct sockaddr { AF_INET, xxx.xx.xxx.xx:yyyyy }
3058 myapp GIO fd 5 wrote 99 bytes
"\M-3\M-l7X{PM[\M^[\M^S\M-Rv:\M^C2\^_=-?\M^[\M-p\M-Kq;/0\M^_\M-`\M-Yx\M-B_\^CyX\M-B\M^?\M^N\M^H\M-[ykO\M-d\M-qg\M-]\M^L\M-BulvAF\M-"\M-d%)\M-Y\M-mK*`\M^L\M^B\M-I\
\M-5\^B\M-/<\M-?\0005\^Y3\M^?q\M-n\M^Ui\M-^\M^?\M^X!\M-w\^RA\M^T\M-|\M-e\M-Fk-\M^P\M-u}\M-j\M^U="
3058 myapp RET sendto 99/0x63
3058 myapp CALL kevent(3,0,0,0x168a7f0c000,128,0)
3058 myapp STRU struct kevent { ident=4, filter=EVFILT_READ, flags=0x61<EV_ADD|EV_CLEAR|EV_RECEIPT>, fflags=0<>, data=69, udata=0x2 }
3058 myapp RET kevent 1
3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0)
3058 myapp GIO fd 4 read 69 bytes
"\0\0\0\^BE\0\0AHN\0\0@\^Q\^Q\M-1\M-@\M-(_\^D\^A\0\0\^A\M-u\M-D\0005\0-\M-w\r{\M-{\^A\0\0\^A\0\0\0\0\0\^A\^Dpow7\^Ccom\0\0\^A\0\^A\0\0)\^D\M-P\0\0\M^@\0\0\0"
3058 myapp RET read 69/0x45
3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0)
3058 myapp GIO fd 4 read 79 bytes
"\0\0\0\^BE\0\0K\M-{\M-3\0\0@\^Q^A\M-@\M-(_\^D\^A\0\0\^A\M-U\M^F\0005\0007\^V\M-J\M^C\M^L\^A\0\0\^A\0\0\0\0\0\^A\atracker\^Fcity9x\^Ccom\0\0\^A\0\^A\0\0)\^D\M-P\0\0\
\M^@\0\0\0"
3058 myapp RET read 79/0x4f
3058 myapp CALL read(4,0x7f7fffff0af0,0x7d0)
3058 myapp RET read -1 errno 35 Resource temporarily unavailable
3058 myapp CALL read(7,0x7f7fffff07e0,0xc)
3058 myapp GIO fd 7 read 12 bytes
"\M^Xr\M-r\M-[%\M-n\\\M^L\M-e&\M^_\M-i"
3058 myapp RET read 12/0xc
3058 myapp CALL sendto(5,0x1689f5f6500,0x5d,0x400<MSG_NOSIGNAL>,0x7f7fffff1144,0x10)
3058 myapp STRU struct sockaddr { AF_INET, xxx.xx.xxx.xx:yyyyy }
3058 myapp RET sendto -1 errno 13 Permission denied
3058 myapp CALL close(5)
3058 myapp RET close 0
3058 myapp CALL munmap(0x1688d3c8000,0x1a1000)
3058 myapp RET munmap 0
3058 myapp CALL munmap(0x169192ad000,0x1a1000)
3058 myapp RET munmap 0
3058 myapp CALL munmap(0x1691cb22000,0x1a1000)
3058 myapp RET munmap 0
3058 myapp CALL munmap(0x168220f8000,0x1a1000)
3058 myapp RET munmap 0
3058 myapp CALL close(4)
3058 myapp RET close 0
3058 myapp CALL close(3)
3058 myapp RET close 0
Siegfried
siegfried.levin@gmail.com
> On Jun 22, 2021, at 3:20 PM, Philip Guenther <guenther@gmail.com> wrote:
>
> On Mon, Jun 21, 2021 at 9:07 PM Siegfried Levin <siegfried.levin@gmail.com <mailto:siegfried.levin@gmail.com>> wrote:
> Thanks a lot for the hint. Unfortunately I'm still not able to see why sendto failed with 13 Permission denied. The AF_INET address masked is the correct one of my server, not a broadcast address. A sendto before this one to the same address just worked.
>
> 3058 myapp CALL sendto(5,0x1689f5f6500,0x5d,0x400<MSG_NOSIGNAL>,0x7f7fffff1144,0x10)
> 3058 myapp STRU struct sockaddr { AF_INET, xxx.xxx.xxx.xxx:yyyy }
>
> Why have you chosen to hide information that may be useful in debugging your problem?
>
> "Hi, I'm asking for help but I have to hide addresses because...this application is insecure if anyone else has its IP+port? Because I've never heard of shodan and don't believe that people are constantly scanning the Internet? And while I don't know why it's failing I'm 1000% sure that there's no information to be gained from seeing the IP, so if it later turns out my understanding of 'broadcast address' is incorrect, the time I've wasted for myself and others will be...a total loss?"
>
>
> 3058 myapp RET sendto -1 errno 13 Permission denied
> 3058 myapp CALL close(5)
> 3058 myapp RET close 0
> The dump file is like 600MB. I can provide more trace log if it is necessary for locating the root cause.
>
> Use the scientific method:
> * make a testable hypothesis
> * devise a test for that
> * perform the test
> * determine whether the hypothesis has been ruled out or confirmed
>
> So, since the manpage mentions blocking pf, I suggest the hypothesis "it returns EACCES because pf is blocking your packets". I can think of several ways to test that; what testing have you performed to confirm or rule out that possibility? "doas pfctl -d; run test; doas pfctl -e"?
>
>
> Alternatively: what's different about *that* call? Does every sento() call on that socket fail? What is special about that socket? If other sendto() calls succeed, what is different about that call? Earlier setsockopt() calls?
>
>
> You say "I can confirm the packet was not sent to a broadcast address": *how* have you confirmed that your understanding of 'broadcast address' matches the kernel's understanding? It ain't just 255.255.255.255....
>
>
> Philip Guenther
>
>
>
>
> Siegfried
> siegfried.levin@gmail.com <mailto:siegfried.levin@gmail.com>
>
>
>
>
> > On Jun 15, 2021, at 8:50 PM, Theo de Raadt <deraadt@openbsd.org <mailto:deraadt@openbsd.org>> wrote:
> >
> > use ktrace
> >
> > Siegfried Levin <siegfried.levin@gmail.com <mailto:siegfried.levin@gmail.com>> wrote:
> >
> >> Hi,
> >>
> >> I have a application run by a normal user communicating with the server with UDP. It crashes very occasionally, like once per week, due to EACCES when sending a UDP packet. According to the manpage (https://man.openbsd.org/OpenBSD-6.9/sendmsg.2#EACCES <https://man.openbsd.org/OpenBSD-6.9/sendmsg.2#EACCES>), the reason might be either being blocked by PF or sending to a broadcast address. I can confirm the packet was not sent to a broadcast address. However, I cannot figure out what rule could block the connection occasionally either. The application can be brought back online without changing any configuration. Does anyone know what might fix this? I can also rewrite the code to make it ignore the error and keep trying but that might not be a proper solution. Running it as root might not be a good idea, too.
> >>
> >> It happens since OpenBSD 6.8. Now I'm running it on 6.9. The application is written in Rust.
> >>
> >> Siegfried
> >> siegfried.levin@gmail.com <mailto:siegfried.levin@gmail.com>
> >>
> >>
> >>
> >>
>
No comments:
Post a Comment