Sunday, June 29, 2025

Re: Ping Spikes

I applied the patch and compiled the kernel but it make no difference.

If you want me to do further testing I am now set up to do that a bit
more quickly.

Mark.


On 6/24/25 07:54, David Gwynne wrote:
>
>
> On 24/06/2025 16:39, Mark de Vries wrote:
>> Will be the first time I apply a patch and compile BSD from source but
>> will have a go. I suppose this is on the 'current' tree?
>
> it should apply to stable too. this code hasn't changed in a while.
>
>>
>> Mark.
>>
>>
>>
>> On 6/24/25 07:19, David Gwynne wrote:
>>> On Mon, Jun 23, 2025 at 03:23:27PM +0100, Mark de Vries wrote:
>>>> Greetings,
>>>>
>>>> I am also seeing increased latency and jitter using an Octeon system -
>>>> Edgerouter 6p. It's the same issue for both OpenBSD 7.6 and 7.7. I
>>>> don't see
>>>> it on a amd64 machine with the same configuration.
>>>>
>>>> Attached is a graph showing on the left ping data collected with
>>>> traffic
>>>> going through a PC-Engines APU1 board with OpenBSD 7.7. To the right is
>>>> traffic going through an Edgerouter 6p with OpenBSD 7.6 . It's the
>>>> same for
>>>> OpenBSD 7.7.
>>>>
>>>> In my case the config is with OpenBGPD on a vlan. This data is
>>>> through NAT
>>>> but it's the same without NAT and it's also the same using PPPoE
>>>> rather than
>>>> BGPD.
>>>>
>>>>  From what I've read on the list it has been observed for Octeon,
>>>> and also
>>>> occasionally on other architectures. As my data shows there are
>>>> cases where
>>>> the amd64 architecture performs as expected.
>>>>
>>>> Is it time for a bug report?
>>>>
>>>> Here is some further info on my system, though note it's 7.6 at the
>>>> moment,
>>>> which I put on temporarily to check if it is unique to version 7.7,
>>>> which it
>>>> isn't.
>>>>
>>>>
>>>>> pfctl -si
>>>>
>>>> Status: Enabled for 0 days 01:20:57              Debug: err
>>>>
>>>> State Table                          Total             Rate
>>>>    current entries                       10
>>>>    half-open tcp                          0
>>>>    searches                           67352           13.9/s
>>>>    inserts                             1262            0.3/s
>>>>    removals                            1252            0.3/s
>>>> Counters
>>>>    match                              15580            3.2/s
>>>>    bad-offset                             0            0.0/s
>>>>    fragment                               0            0.0/s
>>>>    short                                  0            0.0/s
>>>>    normalize                              4            0.0/s
>>>>    memory                                 0            0.0/s
>>>>    bad-timestamp                          0            0.0/s
>>>>    congestion                             0            0.0/s
>>>>    ip-option                              0            0.0/s
>>>>    proto-cksum                            0            0.0/s
>>>>    state-mismatch                        22            0.0/s
>>>>    state-insert                           0            0.0/s
>>>>    state-limit                            0            0.0/s
>>>>    src-limit                              1            0.0/s
>>>>    synproxy                               0            0.0/s
>>>>    translate                              0            0.0/s
>>>>    no-route                               0            0.0/s
>>>>
>>>>> netstat -m
>>>> 1551 mbufs in use:
>>>>          1538 mbufs allocated to data
>>>>          5 mbufs allocated to packet headers
>>>>          8 mbufs allocated to socket names and addresses
>>>> 1536/1712 mbuf 2048 byte clusters in use (current/peak)
>>>> 0/0 mbuf 2112 byte clusters in use (current/peak)
>>>> 0/40 mbuf 4096 byte clusters in use (current/peak)
>>>> 0/24 mbuf 8192 byte clusters in use (current/peak)
>>>> 0/0 mbuf 9216 byte clusters in use (current/peak)
>>>> 0/0 mbuf 12288 byte clusters in use (current/peak)
>>>> 0/0 mbuf 16384 byte clusters in use (current/peak)
>>>> 0/8 mbuf 65536 byte clusters in use (current/peak)
>>>> 4672/4720/131072 Kbytes allocated to network (current/peak/max)
>>>> 0 requests for memory denied
>>>> 0 requests for memory delayed
>>>> 0 calls to protocol drain routines
>>>> 0 defrag mbuf allocation
>>>> 1001 prepend mbuf allocation
>>>> 0 pullup mbuf allocation
>>>> 0 pullup memory copy
>>>> 0 pulldown mbuf allocation
>>>> 0 pulldown memory copy
>>>>
>>>>> systat vm
>>>>
>>>>
>>>>    2 users Load 0.00 0.00 0.00                        15:14:23
>>>>
>>>>              memory totals (in KB)            PAGING SWAPPING
>>>> Interrupts
>>>>             real   virtual     free           in  out   in out      576
>>>> total
>>>> Active    95280     95280   630160
>>>> ops                                soft
>>>> All      380976    380976   841072 pages                          528
>>>> clock
>>>> 3
>>>> cnmac1
>>>> Proc:r  d  s  w    Csw   Trp   Sys   Int   Sof  Flt forks       6
>>>> cnmac2
>>>>            58        57     7    63   576   100   13 fkppw com0
>>>> fksvm octmmc0
>>>>     0.0%Int   0.0%Spn   0.1%Sys   0.0%Usr  99.9%Idle pwait      39 ipi
>>>> |    |    |    |    |    |    |    |    |    |    | relck
>>>> rlkok
>>>> noram
>>>> Namei         Sys-cache    Proc-cache    No-cache       1 ndcpy
>>>>      Calls     hits    %    hits     %    miss   % fltcp
>>>>         10       10  100                                 2 zfod
>>>>                                                            cow
>>>> Disks   sd0                                          2106 fmin
>>>> seeks                                                2808 ftarg
>>>> xfers itarg
>>>> speed                                                   2 wired
>>>>    sec pdfre
>>>> pdscn
>>>> pzidl      14
>>>> IPKTS
>>>>                                                          8
>>>> kmape      12
>>>> OPKTS
>>>>
>>>>> systat mbufs
>>>>
>>>>
>>>> cnmac4   2 users Load 0.00 0.00 0.00 15:15:15
>>>>
>>>> IFACE            RING      LIVELOCKS  SIZE ALIVE   LWM   HWM CWM
>>>> System           mbufs             0   256  1551          27
>>>>                   mcl2k                2048  1536         214
>>>>                   mcl4k                4096     0           5
>>>>                   mcl8k                8192     0           3
>>>>                   mcl64k              65536     0           1
>>>> lo0
>>>> cnmac0
>>>> cnmac1
>>>> cnmac2
>>>> cnmac3
>>>> cnmac4
>>>> cnmac5
>>>> enc0
>>>> vether0
>>>> vlan100
>>>>
>>>> dmesg:
>>>> root@hrr67:~$ dmesg
>>>> [ using 773576 bytes of bsd ELF symbol table ]
>>>> Copyright (c) 1982, 1986, 1989, 1991, 1993
>>>>          The Regents of the University of California.  All rights
>>>> reserved.
>>>> Copyright (c) 1995-2024 OpenBSD. All rights reserved.
>>>> https://www.OpenBSD.org
>>>>
>>>> OpenBSD 7.6 (GENERIC.MP) #233: Mon Sep 30 09:41:48 MDT 2024
>>>> deraadt@octeon.openbsd.org:/usr/src/sys/arch/octeon/compile/GENERIC.MP
>>>> real mem = 1073741824 (1024MB)
>>>> avail mem = 1035321344 (987MB)
>>>> random: good seed from bootblocks
>>>> mainbus0 at root: board 20300 rev 1.23, model cavium,ubnt_e300
>>>> cpu0 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xx
>>>> FPU rev
>>>> 0.0
>>>> cpu0: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
>>>> cpu1 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xx
>>>> FPU rev
>>>> 0.0
>>>> cpu1: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
>>>> cpu2 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xx
>>>> FPU rev
>>>> 0.0
>>>> cpu2: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
>>>> cpu3 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz, CN70xx/CN71xx
>>>> FPU rev
>>>> 0.0
>>>> cpu3: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
>>>> clock0 at mainbus0: int 5
>>>> octcrypto0 at mainbus0
>>>> iobus0 at mainbus0
>>>> simplebus0 at iobus0: "soc"
>>>> "bootbus" at simplebus0 not configured
>>>> octciu0 at simplebus0
>>>> octcib0 at simplebus0: max-bits 23
>>>> octcib1 at simplebus0: max-bits 12
>>>> octcib2 at simplebus0: max-bits 6
>>>> octcib3 at simplebus0: max-bits 15
>>>> octcib4 at simplebus0: max-bits 4
>>>> octcib5 at simplebus0: max-bits 11
>>>> octcib6 at simplebus0: max-bits 11
>>>> octgpio0 at simplebus0: 20 pins, xbit 16
>>>> octsmi0 at simplebus0
>>>> octsmi1 at simplebus0
>>>> octpip0 at simplebus0
>>>> octgmx0 at octpip0 interface 0
>>>> cnmac0 at octgmx0: port 0 SGMII, address 18:e8:29:ba:19:1b
>>>> ukphy0 at cnmac0 phy 4: Generic IEEE 802.3u media interface, rev. 2:
>>>> OUI
>>>> 0x0001c1, model 0x000c
>>>> cnmac1 at octgmx0: port 1 SGMII, address 18:e8:29:ba:19:1c
>>>> ukphy1 at cnmac1 phy 5: Generic IEEE 802.3u media interface, rev. 2:
>>>> OUI
>>>> 0x0001c1, model 0x000c
>>>> cnmac2 at octgmx0: port 2 SGMII, address 18:e8:29:ba:19:1d
>>>> ukphy2 at cnmac2 phy 6: Generic IEEE 802.3u media interface, rev. 2:
>>>> OUI
>>>> 0x0001c1, model 0x000c
>>>> cnmac3 at octgmx0: port 3 SGMII, address 18:e8:29:ba:19:1e
>>>> ukphy3 at cnmac3 phy 7: Generic IEEE 802.3u media interface, rev. 2:
>>>> OUI
>>>> 0x0001c1, model 0x000c
>>>> octgmx1 at octpip0 interface 1
>>>> cnmac4 at octgmx1: port 16 SGMII, address 18:e8:29:ba:19:1f
>>>> ukphy4 at cnmac4 phy 8: Generic IEEE 802.3u media interface, rev. 0:
>>>> OUI
>>>> 0x0001c1, model 0x0027
>>>> cnmac5 at octgmx1: port 17 SGMII, address 18:e8:29:ba:19:20
>>>> ukphy5 at cnmac5 phy 9: Generic IEEE 802.3u media interface, rev. 0:
>>>> OUI
>>>> 0x0001c1, model 0x0027
>>>> octsctl0 at simplebus0: disabled
>>>> octxctl0 at simplebus0: DWC3 rev 0x250a
>>>> xhci0 at octxctl0, xHCI 1.0
>>>> usb0 at xhci0: USB revision 3.0
>>>> uhub0 at usb0 configuration 1 interface 0 "Generic xHCI root hub" rev
>>>> 3.00/1.00 addr 1
>>>> octxctl1 at simplebus0: DWC3 rev 0x250a
>>>> xhci1 at octxctl1, xHCI 1.0
>>>> usb1 at xhci1: USB revision 3.0
>>>> uhub1 at usb1 configuration 1 interface 0 "Generic xHCI root hub" rev
>>>> 3.00/1.00 addr 1
>>>> "i2c" at simplebus0 not configured
>>>> "i2c" at simplebus0 not configured
>>>> com0 at simplebus0: ns16550a, 64 byte fifo
>>>> com0: console
>>>> com1 at simplebus0: ns16550a, 64 byte fifo
>>>> com1: probed fifo depth: 0 bytes
>>>> octmmc0 at simplebus0
>>>> sdmmc0 at octmmc0: 8-bit, mmc high-speed
>>>> "spi" at simplebus0 not configured
>>>> "ocla0" at simplebus0 not configured
>>>> "dma-engine" at simplebus0 not configured
>>>> "dma-engine" at simplebus0 not configured
>>>> octrng0 at iobus0 base 0x1400000000000 irq 0
>>>> octpcie0 at iobus0: 3 ports
>>>> octpcie0 port 0: link timeout
>>>> octpcie0 port 1: reset timeout
>>>> octpcie0 port 2: reset timeout
>>>> scsibus0 at sdmmc0: 2 targets, initiator 0
>>>> sd0 at scsibus0 targ 1 lun 0: <Kingston, MMC4GB, 0000> removable
>>>> sd0: 3728MB, 512 bytes/sector, 7634944 sectors
>>>> vscsi0 at root
>>>> scsibus1 at vscsi0: 256 targets
>>>> softraid0 at root
>>>> scsibus2 at softraid0: 256 targets
>>>> root on sd0a (285e489508f0f02b.a) swap on sd0b dump on sd0b
>>>
>>> can you try this?
>>>
>>> Index: uipc_socket.c
>>> ===================================================================
>>> RCS file: /cvs/src/sys/kern/uipc_socket.c,v
>>> diff -u -p -r1.344 uipc_socket.c
>>> --- uipc_socket.c    31 Oct 2024 12:51:55 -0000    1.344
>>> +++ uipc_socket.c    1 Nov 2024 23:53:25 -0000
>>> @@ -787,48 +787,36 @@ m_getuio(struct mbuf **mp, int atomic, l
>>>   {
>>>       struct mbuf *m, *top = NULL;
>>>       struct mbuf **nextp = &top;
>>> -    u_long len, mlen;
>>> -    size_t resid = uio->uio_resid;
>>> +    u_long len, mlen, alen;
>>> +    int align = atomic ? roundup(max_hdr, sizeof(long)) : 0;
>>>       int error;
>>>   -    do {
>>> -        if (top == NULL) {
>>> -            MGETHDR(m, M_WAIT, MT_DATA);
>>> -            mlen = MHLEN;
>>> -        } else {
>>> -            MGET(m, M_WAIT, MT_DATA);
>>> -            mlen = MLEN;
>>> -        }
>>> +    m = m_gethdr(M_WAIT, MT_DATA);
>>> +    mlen = MHLEN;
>>> +
>>> +    for (;;) {
>>>           /* chain mbuf together */
>>>           *nextp = m;
>>>           nextp = &m->m_next;
>>>   -        resid = ulmin(resid, space);
>>> -        if (resid >= MINCLSIZE) {
>>> -            MCLGETL(m, M_NOWAIT, ulmin(resid, MAXMCLBYTES));
>>> -            if ((m->m_flags & M_EXT) == 0)
>>> +        /* How much data we want to put in this mbuf? */
>>> +        len = ulmin(uio->uio_resid, space);
>>> +        /* How much space are we allocating for that data? */
>>> +        alen = align + len;
>>> +        if (alen > mlen) {
>>> +            MCLGETL(m, M_NOWAIT, ulmin(alen, MAXMCLBYTES));
>>> +            if (!ISSET(m->m_flags, M_EXT) && alen > MCLBYTES)
>>>                   MCLGETL(m, M_NOWAIT, MCLBYTES);
>>> -            if ((m->m_flags & M_EXT) == 0)
>>> -                goto nopages;
>>> -            mlen = m->m_ext.ext_size;
>>> -            len = ulmin(mlen, resid);
>>> -            /*
>>> -             * For datagram protocols, leave room
>>> -             * for protocol headers in first mbuf.
>>> -             */
>>> -            if (atomic && m == top && len < mlen - max_hdr)
>>> -                m->m_data += max_hdr;
>>> -        } else {
>>> -nopages:
>>> -            len = ulmin(mlen, resid);
>>> -            /*
>>> -             * For datagram protocols, leave room
>>> -             * for protocol headers in first mbuf.
>>> -             */
>>> -            if (atomic && m == top && len < mlen - max_hdr)
>>> -                m_align(m, len);
>>> +            if (ISSET(m->m_flags, M_EXT))
>>> +                mlen = m->m_ext.ext_size;
>>>           }
>>>   +        /* Avoid pain from a stupid max_hdr value */
>>> +        if (align < mlen)
>>> +            m->m_data += align;
>>> +
>>> +        /* How much data can we put in this mbuf? */
>>> +        len = ulmin(mlen, len);
>>>           error = uiomove(mtod(m, caddr_t), len, uio);
>>>           if (error) {
>>>               m_freem(top);
>>> @@ -836,13 +824,19 @@ nopages:
>>>           }
>>>             /* adjust counters */
>>> -        resid = uio->uio_resid;
>>>           space -= len;
>>>           m->m_len = len;
>>>           top->m_pkthdr.len += len;
>>>             /* Is there more space and more data? */
>>> -    } while (space > 0 && resid > 0);
>>> +        if (space == 0 || uio->uio_resid == 0)
>>> +            break;
>>> +
>>> +        align = 0;
>>> +
>>> +        m = m_get(M_WAIT, MT_DATA);
>>> +        mlen = MLEN;
>>> +    }
>>>         *mp = top;
>>>       return 0;
>>>
>>
>>
>
>

No comments:

Post a Comment