On 24/06/2025 16:39, Mark de Vries wrote:
> Will be the first time I apply a patch and compile BSD from source but
> will have a go. I suppose this is on the 'current' tree?
it should apply to stable too. this code hasn't changed in a while.
>
> Mark.
>
>
>
> On 6/24/25 07:19, David Gwynne wrote:
>> On Mon, Jun 23, 2025 at 03:23:27PM +0100, Mark de Vries wrote:
>>> Greetings,
>>>
>>> I am also seeing increased latency and jitter using an Octeon system -
>>> Edgerouter 6p. It's the same issue for both OpenBSD 7.6 and 7.7. I
>>> don't see
>>> it on a amd64 machine with the same configuration.
>>>
>>> Attached is a graph showing on the left ping data collected with
>>> traffic
>>> going through a PC-Engines APU1 board with OpenBSD 7.7. To the right is
>>> traffic going through an Edgerouter 6p with OpenBSD 7.6 . It's the
>>> same for
>>> OpenBSD 7.7.
>>>
>>> In my case the config is with OpenBGPD on a vlan. This data is
>>> through NAT
>>> but it's the same without NAT and it's also the same using PPPoE
>>> rather than
>>> BGPD.
>>>
>>> From what I've read on the list it has been observed for Octeon,
>>> and also
>>> occasionally on other architectures. As my data shows there are
>>> cases where
>>> the amd64 architecture performs as expected.
>>>
>>> Is it time for a bug report?
>>>
>>> Here is some further info on my system, though note it's 7.6 at the
>>> moment,
>>> which I put on temporarily to check if it is unique to version 7.7,
>>> which it
>>> isn't.
>>>
>>>
>>>> pfctl -si
>>>
>>> Status: Enabled for 0 days 01:20:57 Debug: err
>>>
>>> State Table Total Rate
>>> current entries 10
>>> half-open tcp 0
>>> searches 67352 13.9/s
>>> inserts 1262 0.3/s
>>> removals 1252 0.3/s
>>> Counters
>>> match 15580 3.2/s
>>> bad-offset 0 0.0/s
>>> fragment 0 0.0/s
>>> short 0 0.0/s
>>> normalize 4 0.0/s
>>> memory 0 0.0/s
>>> bad-timestamp 0 0.0/s
>>> congestion 0 0.0/s
>>> ip-option 0 0.0/s
>>> proto-cksum 0 0.0/s
>>> state-mismatch 22 0.0/s
>>> state-insert 0 0.0/s
>>> state-limit 0 0.0/s
>>> src-limit 1 0.0/s
>>> synproxy 0 0.0/s
>>> translate 0 0.0/s
>>> no-route 0 0.0/s
>>>
>>>> netstat -m
>>> 1551 mbufs in use:
>>> 1538 mbufs allocated to data
>>> 5 mbufs allocated to packet headers
>>> 8 mbufs allocated to socket names and addresses
>>> 1536/1712 mbuf 2048 byte clusters in use (current/peak)
>>> 0/0 mbuf 2112 byte clusters in use (current/peak)
>>> 0/40 mbuf 4096 byte clusters in use (current/peak)
>>> 0/24 mbuf 8192 byte clusters in use (current/peak)
>>> 0/0 mbuf 9216 byte clusters in use (current/peak)
>>> 0/0 mbuf 12288 byte clusters in use (current/peak)
>>> 0/0 mbuf 16384 byte clusters in use (current/peak)
>>> 0/8 mbuf 65536 byte clusters in use (current/peak)
>>> 4672/4720/131072 Kbytes allocated to network (current/peak/max)
>>> 0 requests for memory denied
>>> 0 requests for memory delayed
>>> 0 calls to protocol drain routines
>>> 0 defrag mbuf allocation
>>> 1001 prepend mbuf allocation
>>> 0 pullup mbuf allocation
>>> 0 pullup memory copy
>>> 0 pulldown mbuf allocation
>>> 0 pulldown memory copy
>>>
>>>> systat vm
>>>
>>>
>>> 2 users Load 0.00 0.00 0.00 15:14:23
>>>
>>> memory totals (in KB) PAGING SWAPPING
>>> Interrupts
>>> real virtual free in out in out 576
>>> total
>>> Active 95280 95280 630160
>>> ops soft
>>> All 380976 380976 841072 pages 528
>>> clock
>>>
>>> 3
>>> cnmac1
>>> Proc:r d s w Csw Trp Sys Int Sof Flt forks 6
>>> cnmac2
>>> 58 57 7 63 576 100 13 fkppw com0
>>> fksvm octmmc0
>>> 0.0%Int 0.0%Spn 0.1%Sys 0.0%Usr 99.9%Idle pwait 39 ipi
>>> | | | | | | | | | | | relck
>>> rlkok
>>> noram
>>> Namei Sys-cache Proc-cache No-cache 1 ndcpy
>>> Calls hits % hits % miss % fltcp
>>> 10 10 100 2 zfod
>>> cow
>>> Disks sd0 2106 fmin
>>> seeks 2808 ftarg
>>> xfers itarg
>>> speed 2 wired
>>> sec pdfre
>>> pdscn
>>> pzidl 14
>>> IPKTS
>>> 8
>>> kmape 12
>>> OPKTS
>>>
>>>> systat mbufs
>>>
>>>
>>> cnmac4 2 users Load 0.00 0.00 0.00 15:15:15
>>>
>>> IFACE RING LIVELOCKS SIZE ALIVE LWM HWM CWM
>>> System mbufs 0 256 1551 27
>>> mcl2k 2048 1536 214
>>> mcl4k 4096 0 5
>>> mcl8k 8192 0 3
>>> mcl64k 65536 0 1
>>> lo0
>>> cnmac0
>>> cnmac1
>>> cnmac2
>>> cnmac3
>>> cnmac4
>>> cnmac5
>>> enc0
>>> vether0
>>> vlan100
>>>
>>> dmesg:
>>> root@hrr67:~$ dmesg
>>> [ using 773576 bytes of bsd ELF symbol table ]
>>> Copyright (c) 1982, 1986, 1989, 1991, 1993
>>> The Regents of the University of California. All rights
>>> reserved.
>>> Copyright (c) 1995-2024 OpenBSD. All rights reserved.
>>> https://www.OpenBSD.org
>>>
>>> OpenBSD 7.6 (GENERIC.MP) #233: Mon Sep 30 09:41:48 MDT 2024
>>> deraadt@octeon.openbsd.org:/usr/src/sys/arch/octeon/compile/GENERIC.MP
>>> real mem = 1073741824 (1024MB)
>>> avail mem = 1035321344 (987MB)
>>> random: good seed from bootblocks
>>> mainbus0 at root: board 20300 rev 1.23, model cavium,ubnt_e300
>>> cpu0 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xx
>>> FPU rev
>>> 0.0
>>> cpu0: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
>>> cpu1 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xx
>>> FPU rev
>>> 0.0
>>> cpu1: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
>>> cpu2 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xx
>>> FPU rev
>>> 0.0
>>> cpu2: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
>>> cpu3 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xx
>>> FPU rev
>>> 0.0
>>> cpu3: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
>>> clock0 at mainbus0: int 5
>>> octcrypto0 at mainbus0
>>> iobus0 at mainbus0
>>> simplebus0 at iobus0: "soc"
>>> "bootbus" at simplebus0 not configured
>>> octciu0 at simplebus0
>>> octcib0 at simplebus0: max-bits 23
>>> octcib1 at simplebus0: max-bits 12
>>> octcib2 at simplebus0: max-bits 6
>>> octcib3 at simplebus0: max-bits 15
>>> octcib4 at simplebus0: max-bits 4
>>> octcib5 at simplebus0: max-bits 11
>>> octcib6 at simplebus0: max-bits 11
>>> octgpio0 at simplebus0: 20 pins, xbit 16
>>> octsmi0 at simplebus0
>>> octsmi1 at simplebus0
>>> octpip0 at simplebus0
>>> octgmx0 at octpip0 interface 0
>>> cnmac0 at octgmx0: port 0 SGMII, address 18:e8:29:ba:19:1b
>>> ukphy0 at cnmac0 phy 4: Generic IEEE 802.3u media interface, rev. 2:
>>> OUI
>>> 0x0001c1, model 0x000c
>>> cnmac1 at octgmx0: port 1 SGMII, address 18:e8:29:ba:19:1c
>>> ukphy1 at cnmac1 phy 5: Generic IEEE 802.3u media interface, rev. 2:
>>> OUI
>>> 0x0001c1, model 0x000c
>>> cnmac2 at octgmx0: port 2 SGMII, address 18:e8:29:ba:19:1d
>>> ukphy2 at cnmac2 phy 6: Generic IEEE 802.3u media interface, rev. 2:
>>> OUI
>>> 0x0001c1, model 0x000c
>>> cnmac3 at octgmx0: port 3 SGMII, address 18:e8:29:ba:19:1e
>>> ukphy3 at cnmac3 phy 7: Generic IEEE 802.3u media interface, rev. 2:
>>> OUI
>>> 0x0001c1, model 0x000c
>>> octgmx1 at octpip0 interface 1
>>> cnmac4 at octgmx1: port 16 SGMII, address 18:e8:29:ba:19:1f
>>> ukphy4 at cnmac4 phy 8: Generic IEEE 802.3u media interface, rev. 0:
>>> OUI
>>> 0x0001c1, model 0x0027
>>> cnmac5 at octgmx1: port 17 SGMII, address 18:e8:29:ba:19:20
>>> ukphy5 at cnmac5 phy 9: Generic IEEE 802.3u media interface, rev. 0:
>>> OUI
>>> 0x0001c1, model 0x0027
>>> octsctl0 at simplebus0: disabled
>>> octxctl0 at simplebus0: DWC3 rev 0x250a
>>> xhci0 at octxctl0, xHCI 1.0
>>> usb0 at xhci0: USB revision 3.0
>>> uhub0 at usb0 configuration 1 interface 0 "Generic xHCI root hub" rev
>>> 3.00/1.00 addr 1
>>> octxctl1 at simplebus0: DWC3 rev 0x250a
>>> xhci1 at octxctl1, xHCI 1.0
>>> usb1 at xhci1: USB revision 3.0
>>> uhub1 at usb1 configuration 1 interface 0 "Generic xHCI root hub" rev
>>> 3.00/1.00 addr 1
>>> "i2c" at simplebus0 not configured
>>> "i2c" at simplebus0 not configured
>>> com0 at simplebus0: ns16550a, 64 byte fifo
>>> com0: console
>>> com1 at simplebus0: ns16550a, 64 byte fifo
>>> com1: probed fifo depth: 0 bytes
>>> octmmc0 at simplebus0
>>> sdmmc0 at octmmc0: 8-bit, mmc high-speed
>>> "spi" at simplebus0 not configured
>>> "ocla0" at simplebus0 not configured
>>> "dma-engine" at simplebus0 not configured
>>> "dma-engine" at simplebus0 not configured
>>> octrng0 at iobus0 base 0x1400000000000 irq 0
>>> octpcie0 at iobus0: 3 ports
>>> octpcie0 port 0: link timeout
>>> octpcie0 port 1: reset timeout
>>> octpcie0 port 2: reset timeout
>>> scsibus0 at sdmmc0: 2 targets, initiator 0
>>> sd0 at scsibus0 targ 1 lun 0: <Kingston, MMC4GB, 0000> removable
>>> sd0: 3728MB, 512 bytes/sector, 7634944 sectors
>>> vscsi0 at root
>>> scsibus1 at vscsi0: 256 targets
>>> softraid0 at root
>>> scsibus2 at softraid0: 256 targets
>>> root on sd0a (285e489508f0f02b.a) swap on sd0b dump on sd0b
>>
>> can you try this?
>>
>> Index: uipc_socket.c
>> ===================================================================
>> RCS file: /cvs/src/sys/kern/uipc_socket.c,v
>> diff -u -p -r1.344 uipc_socket.c
>> --- uipc_socket.c 31 Oct 2024 12:51:55 -0000 1.344
>> +++ uipc_socket.c 1 Nov 2024 23:53:25 -0000
>> @@ -787,48 +787,36 @@ m_getuio(struct mbuf **mp, int atomic, l
>> {
>> struct mbuf *m, *top = NULL;
>> struct mbuf **nextp = ⊤
>> - u_long len, mlen;
>> - size_t resid = uio->uio_resid;
>> + u_long len, mlen, alen;
>> + int align = atomic ? roundup(max_hdr, sizeof(long)) : 0;
>> int error;
>> - do {
>> - if (top == NULL) {
>> - MGETHDR(m, M_WAIT, MT_DATA);
>> - mlen = MHLEN;
>> - } else {
>> - MGET(m, M_WAIT, MT_DATA);
>> - mlen = MLEN;
>> - }
>> + m = m_gethdr(M_WAIT, MT_DATA);
>> + mlen = MHLEN;
>> +
>> + for (;;) {
>> /* chain mbuf together */
>> *nextp = m;
>> nextp = &m->m_next;
>> - resid = ulmin(resid, space);
>> - if (resid >= MINCLSIZE) {
>> - MCLGETL(m, M_NOWAIT, ulmin(resid, MAXMCLBYTES));
>> - if ((m->m_flags & M_EXT) == 0)
>> + /* How much data we want to put in this mbuf? */
>> + len = ulmin(uio->uio_resid, space);
>> + /* How much space are we allocating for that data? */
>> + alen = align + len;
>> + if (alen > mlen) {
>> + MCLGETL(m, M_NOWAIT, ulmin(alen, MAXMCLBYTES));
>> + if (!ISSET(m->m_flags, M_EXT) && alen > MCLBYTES)
>> MCLGETL(m, M_NOWAIT, MCLBYTES);
>> - if ((m->m_flags & M_EXT) == 0)
>> - goto nopages;
>> - mlen = m->m_ext.ext_size;
>> - len = ulmin(mlen, resid);
>> - /*
>> - * For datagram protocols, leave room
>> - * for protocol headers in first mbuf.
>> - */
>> - if (atomic && m == top && len < mlen - max_hdr)
>> - m->m_data += max_hdr;
>> - } else {
>> -nopages:
>> - len = ulmin(mlen, resid);
>> - /*
>> - * For datagram protocols, leave room
>> - * for protocol headers in first mbuf.
>> - */
>> - if (atomic && m == top && len < mlen - max_hdr)
>> - m_align(m, len);
>> + if (ISSET(m->m_flags, M_EXT))
>> + mlen = m->m_ext.ext_size;
>> }
>> + /* Avoid pain from a stupid max_hdr value */
>> + if (align < mlen)
>> + m->m_data += align;
>> +
>> + /* How much data can we put in this mbuf? */
>> + len = ulmin(mlen, len);
>> error = uiomove(mtod(m, caddr_t), len, uio);
>> if (error) {
>> m_freem(top);
>> @@ -836,13 +824,19 @@ nopages:
>> }
>> /* adjust counters */
>> - resid = uio->uio_resid;
>> space -= len;
>> m->m_len = len;
>> top->m_pkthdr.len += len;
>> /* Is there more space and more data? */
>> - } while (space > 0 && resid > 0);
>> + if (space == 0 || uio->uio_resid == 0)
>> + break;
>> +
>> + align = 0;
>> +
>> + m = m_get(M_WAIT, MT_DATA);
>> + mlen = MLEN;
>> + }
>> *mp = top;
>> return 0;
>>
>
>
No comments:
Post a Comment