Hi,
I'm just putting this out there "just in case". I had ping spikes on my
Protectli box. This thread lead to a resolution for me, but not a happy
one (disable inteldrm). I bought an HDMI "dummy plug" and that resolved
it without inteldrm being disabled.
It makes absolutely no sense that the HDMI could impact the latency, but
it 100% does and is 100% repeatable.
https://www.reddit.com/r/openbsd/comments/105c0zk/comment/jg4aq13/
I did a "dmesg" before and after the dummy plug and re-enabling inteldrm.
mini$ diff dmesg_after_recompile.txt dmesg_after_dummy.txt
4c4
< avail mem = 8234815488 (7853MB)
---
> avail mem = 8234778624 (7853MB)
21c21
< cpu0: Intel(R) Celeron(R) CPU J3060 @ 1.60GHz, 2480.27 MHz, 06-4c-04,
patch 00000411
---
> cpu0: Intel(R) Celeron(R) CPU J3060 @ 1.60GHz, 2480.28 MHz, 06-4c-04,
patch 00000411
35c35
< cpu1: Intel(R) Celeron(R) CPU J3060 @ 1.60GHz, 2480.46 MHz, 06-4c-04,
patch 00000411
---
> cpu1: Intel(R) Celeron(R) CPU J3060 @ 1.60GHz, 2481.05 MHz, 06-4c-04,
patch 00000411
56,58c56,58
< inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics" rev 0x35
< drm0 at inteldrm0
< inteldrm0: msi, CHERRYVIEW, gen 8
---
> vga1 at pci0 dev 2 function 0 "Intel HD Graphics" rev 0x35
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
160,162d159
< inteldrm0: 1920x1080, 32bpp
< wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation)
< wsdisplay0: screen 1-5 added (std, vt100 emulation)
On 10/30/2025 8:43 AM, Mark de Vries wrote:
> There have been at least two separate issues being flagged up in this
> thread. At least
> one of the issues initially reported have since been resolved/explained.
>
> Meanwhile others including myself have posted about ping spikes for
> the Octeon
> architecture, as observed on Ubiquity Edgerouters including the ER-6p
> and ER-Lite.
>
> A patch was send out and I have tried this but it didn't make a
> difference. As another
> test I compared OpenBSD-78 and FreeBSD 11 (from 2017 because the
> Octeon architecture
> is no longer supported on FreeBSD).
>
> Here are the ping stats with the same Edgerouter Lite in the same test
> setup using
> Smokeping:
>
> Edgerouter Lite running FreeBSD 11: avg 436 us, max 453 us, min 409
> us, std 0 us.
> Same box running OpenBSD 7.8: avg 1.9 ms, max 4.5 ms, min 578
> ms, std 1.2 ms.
>
> So on the old FreeBSD install the router works as it should. Running
> OpenBSD it would
> not be suitable as a network device, it would add too much latency and
> jitter.
>
> Much of the appeal of the hardware with Octeon processors is that they
> are ideal as
> network devices. I also think OpenBSD is an ideal platform for routers
> of any
> size. So it is unfortunate that the packet processing/forwarding
> capability here is
> compromised.
>
> I thought I try FreeBSD to compare with, in the hope that the code
> bases are
> sufficiently similar so that the FreeBSD code could point us in the
> right direction
> to solve this issue on OpenBSD.
>
> If anyone can have another look at this that would be greatly
> appreciated.
>
> Mark de Vries.
>
> On 02-07-2025 22:15, David Gwynne wrote:
>>> On 30 Jun 2025, at 05:00, Mark de Vries <mark@hubs.net.uk> wrote:
>>>
>>> I applied the patch and compiled the kernel but it make no difference.
>>
>> Cool, good to know. I guess I should get a test setup going.
>>
>>>
>>> If you want me to do further testing I am now set up to do that a
>>> bit more quickly.
>>>
>>> Mark.
>>>
>>>
>>> On 6/24/25 07:54, David Gwynne wrote:
>>>> On 24/06/2025 16:39, Mark de Vries wrote:
>>>>> Will be the first time I apply a patch and compile BSD from source
>>>>> but will have a go. I suppose this is on the 'current' tree?
>>>> it should apply to stable too. this code hasn't changed in a while.
>>>>>
>>>>> Mark.
>>>>>
>>>>>
>>>>>
>>>>> On 6/24/25 07:19, David Gwynne wrote:
>>>>>> On Mon, Jun 23, 2025 at 03:23:27PM +0100, Mark de Vries wrote:
>>>>>>> Greetings,
>>>>>>>
>>>>>>> I am also seeing increased latency and jitter using an Octeon
>>>>>>> system -
>>>>>>> Edgerouter 6p. It's the same issue for both OpenBSD 7.6 and 7.7.
>>>>>>> I don't see
>>>>>>> it on a amd64 machine with the same configuration.
>>>>>>>
>>>>>>> Attached is a graph showing on the left ping data collected with
>>>>>>> traffic
>>>>>>> going through a PC-Engines APU1 board with OpenBSD 7.7. To the
>>>>>>> right is
>>>>>>> traffic going through an Edgerouter 6p with OpenBSD 7.6 . It's
>>>>>>> the same for
>>>>>>> OpenBSD 7.7.
>>>>>>>
>>>>>>> In my case the config is with OpenBGPD on a vlan. This data is
>>>>>>> through NAT
>>>>>>> but it's the same without NAT and it's also the same using PPPoE
>>>>>>> rather than
>>>>>>> BGPD.
>>>>>>>
>>>>>>> From what I've read on the list it has been observed for
>>>>>>> Octeon, and also
>>>>>>> occasionally on other architectures. As my data shows there are
>>>>>>> cases where
>>>>>>> the amd64 architecture performs as expected.
>>>>>>>
>>>>>>> Is it time for a bug report?
>>>>>>>
>>>>>>> Here is some further info on my system, though note it's 7.6 at
>>>>>>> the moment,
>>>>>>> which I put on temporarily to check if it is unique to version
>>>>>>> 7.7, which it
>>>>>>> isn't.
>>>>>>>
>>>>>>>
>>>>>>>> pfctl -si
>>>>>>>
>>>>>>> Status: Enabled for 0 days 01:20:57 Debug: err
>>>>>>>
>>>>>>> State Table Total Rate
>>>>>>> current entries 10
>>>>>>> half-open tcp 0
>>>>>>> searches 67352 13.9/s
>>>>>>> inserts 1262 0.3/s
>>>>>>> removals 1252 0.3/s
>>>>>>> Counters
>>>>>>> match 15580 3.2/s
>>>>>>> bad-offset 0 0.0/s
>>>>>>> fragment 0 0.0/s
>>>>>>> short 0 0.0/s
>>>>>>> normalize 4 0.0/s
>>>>>>> memory 0 0.0/s
>>>>>>> bad-timestamp 0 0.0/s
>>>>>>> congestion 0 0.0/s
>>>>>>> ip-option 0 0.0/s
>>>>>>> proto-cksum 0 0.0/s
>>>>>>> state-mismatch 22 0.0/s
>>>>>>> state-insert 0 0.0/s
>>>>>>> state-limit 0 0.0/s
>>>>>>> src-limit 1 0.0/s
>>>>>>> synproxy 0 0.0/s
>>>>>>> translate 0 0.0/s
>>>>>>> no-route 0 0.0/s
>>>>>>>
>>>>>>>> netstat -m
>>>>>>> 1551 mbufs in use:
>>>>>>> 1538 mbufs allocated to data
>>>>>>> 5 mbufs allocated to packet headers
>>>>>>> 8 mbufs allocated to socket names and addresses
>>>>>>> 1536/1712 mbuf 2048 byte clusters in use (current/peak)
>>>>>>> 0/0 mbuf 2112 byte clusters in use (current/peak)
>>>>>>> 0/40 mbuf 4096 byte clusters in use (current/peak)
>>>>>>> 0/24 mbuf 8192 byte clusters in use (current/peak)
>>>>>>> 0/0 mbuf 9216 byte clusters in use (current/peak)
>>>>>>> 0/0 mbuf 12288 byte clusters in use (current/peak)
>>>>>>> 0/0 mbuf 16384 byte clusters in use (current/peak)
>>>>>>> 0/8 mbuf 65536 byte clusters in use (current/peak)
>>>>>>> 4672/4720/131072 Kbytes allocated to network (current/peak/max)
>>>>>>> 0 requests for memory denied
>>>>>>> 0 requests for memory delayed
>>>>>>> 0 calls to protocol drain routines
>>>>>>> 0 defrag mbuf allocation
>>>>>>> 1001 prepend mbuf allocation
>>>>>>> 0 pullup mbuf allocation
>>>>>>> 0 pullup memory copy
>>>>>>> 0 pulldown mbuf allocation
>>>>>>> 0 pulldown memory copy
>>>>>>>
>>>>>>>> systat vm
>>>>>>>
>>>>>>>
>>>>>>> 2 users Load 0.00 0.00 0.00 15:14:23
>>>>>>>
>>>>>>> memory totals (in KB) PAGING SWAPPING
>>>>>>> Interrupts
>>>>>>> real virtual free in out in
>>>>>>> out 576
>>>>>>> total
>>>>>>> Active 95280 95280 630160
>>>>>>> ops soft
>>>>>>> All 380976 380976 841072
>>>>>>> pages 528
>>>>>>> clock
>>>>>>> 3
>>>>>>> cnmac1
>>>>>>> Proc:r d s w Csw Trp Sys Int Sof Flt forks 6
>>>>>>> cnmac2
>>>>>>> 58 57 7 63 576 100 13 fkppw com0
>>>>>>> fksvm octmmc0
>>>>>>> 0.0%Int 0.0%Spn 0.1%Sys 0.0%Usr 99.9%Idle pwait
>>>>>>> 39 ipi
>>>>>>> | | | | | | | | | | | relck
>>>>>>> rlkok
>>>>>>> noram
>>>>>>> Namei Sys-cache Proc-cache No-cache 1 ndcpy
>>>>>>> Calls hits % hits % miss % fltcp
>>>>>>> 10 10 100 2 zfod
>>>>>>> cow
>>>>>>> Disks sd0 2106 fmin
>>>>>>> seeks 2808 ftarg
>>>>>>> xfers itarg
>>>>>>> speed 2 wired
>>>>>>> sec pdfre
>>>>>>> pdscn
>>>>>>> pzidl 14
>>>>>>> IPKTS
>>>>>>> 8
>>>>>>> kmape 12
>>>>>>> OPKTS
>>>>>>>
>>>>>>>> systat mbufs
>>>>>>>
>>>>>>>
>>>>>>> cnmac4 2 users Load 0.00 0.00 0.00 15:15:15
>>>>>>>
>>>>>>> IFACE RING LIVELOCKS SIZE ALIVE LWM HWM CWM
>>>>>>> System mbufs 0 256 1551 27
>>>>>>> mcl2k 2048 1536 214
>>>>>>> mcl4k 4096 0 5
>>>>>>> mcl8k 8192 0 3
>>>>>>> mcl64k 65536 0 1
>>>>>>> lo0
>>>>>>> cnmac0
>>>>>>> cnmac1
>>>>>>> cnmac2
>>>>>>> cnmac3
>>>>>>> cnmac4
>>>>>>> cnmac5
>>>>>>> enc0
>>>>>>> vether0
>>>>>>> vlan100
>>>>>>>
>>>>>>> dmesg:
>>>>>>> root@hrr67:~$ dmesg
>>>>>>> [ using 773576 bytes of bsd ELF symbol table ]
>>>>>>> Copyright (c) 1982, 1986, 1989, 1991, 1993
>>>>>>> The Regents of the University of California. All rights
>>>>>>> reserved.
>>>>>>> Copyright (c) 1995-2024 OpenBSD. All rights reserved.
>>>>>>> https://www.OpenBSD.org
>>>>>>>
>>>>>>> OpenBSD 7.6 (GENERIC.MP) #233: Mon Sep 30 09:41:48 MDT 2024
>>>>>>> deraadt@octeon.openbsd.org:/usr/src/sys/arch/octeon/compile/GENERIC.MP
>>>>>>>
>>>>>>> real mem = 1073741824 (1024MB)
>>>>>>> avail mem = 1035321344 (987MB)
>>>>>>> random: good seed from bootblocks
>>>>>>> mainbus0 at root: board 20300 rev 1.23, model cavium,ubnt_e300
>>>>>>> cpu0 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz,
>>>>>>> CN70xx/CN71xx FPU rev
>>>>>>> 0.0
>>>>>>> cpu0: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
>>>>>>> cpu1 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz,
>>>>>>> CN70xx/CN71xx FPU rev
>>>>>>> 0.0
>>>>>>> cpu1: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
>>>>>>> cpu2 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz,
>>>>>>> CN70xx/CN71xx FPU rev
>>>>>>> 0.0
>>>>>>> cpu2: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
>>>>>>> cpu3 at mainbus0: CN70xx/CN71xx CPU rev 0.2 1000 MHz,
>>>>>>> CN70xx/CN71xx FPU rev
>>>>>>> 0.0
>>>>>>> cpu3: cache L1-I 78KB 39 way D 32KB 32 way, L2 1024KB 8 way
>>>>>>> clock0 at mainbus0: int 5
>>>>>>> octcrypto0 at mainbus0
>>>>>>> iobus0 at mainbus0
>>>>>>> simplebus0 at iobus0: "soc"
>>>>>>> "bootbus" at simplebus0 not configured
>>>>>>> octciu0 at simplebus0
>>>>>>> octcib0 at simplebus0: max-bits 23
>>>>>>> octcib1 at simplebus0: max-bits 12
>>>>>>> octcib2 at simplebus0: max-bits 6
>>>>>>> octcib3 at simplebus0: max-bits 15
>>>>>>> octcib4 at simplebus0: max-bits 4
>>>>>>> octcib5 at simplebus0: max-bits 11
>>>>>>> octcib6 at simplebus0: max-bits 11
>>>>>>> octgpio0 at simplebus0: 20 pins, xbit 16
>>>>>>> octsmi0 at simplebus0
>>>>>>> octsmi1 at simplebus0
>>>>>>> octpip0 at simplebus0
>>>>>>> octgmx0 at octpip0 interface 0
>>>>>>> cnmac0 at octgmx0: port 0 SGMII, address 18:e8:29:ba:19:1b
>>>>>>> ukphy0 at cnmac0 phy 4: Generic IEEE 802.3u media interface,
>>>>>>> rev. 2: OUI
>>>>>>> 0x0001c1, model 0x000c
>>>>>>> cnmac1 at octgmx0: port 1 SGMII, address 18:e8:29:ba:19:1c
>>>>>>> ukphy1 at cnmac1 phy 5: Generic IEEE 802.3u media interface,
>>>>>>> rev. 2: OUI
>>>>>>> 0x0001c1, model 0x000c
>>>>>>> cnmac2 at octgmx0: port 2 SGMII, address 18:e8:29:ba:19:1d
>>>>>>> ukphy2 at cnmac2 phy 6: Generic IEEE 802.3u media interface,
>>>>>>> rev. 2: OUI
>>>>>>> 0x0001c1, model 0x000c
>>>>>>> cnmac3 at octgmx0: port 3 SGMII, address 18:e8:29:ba:19:1e
>>>>>>> ukphy3 at cnmac3 phy 7: Generic IEEE 802.3u media interface,
>>>>>>> rev. 2: OUI
>>>>>>> 0x0001c1, model 0x000c
>>>>>>> octgmx1 at octpip0 interface 1
>>>>>>> cnmac4 at octgmx1: port 16 SGMII, address 18:e8:29:ba:19:1f
>>>>>>> ukphy4 at cnmac4 phy 8: Generic IEEE 802.3u media interface,
>>>>>>> rev. 0: OUI
>>>>>>> 0x0001c1, model 0x0027
>>>>>>> cnmac5 at octgmx1: port 17 SGMII, address 18:e8:29:ba:19:20
>>>>>>> ukphy5 at cnmac5 phy 9: Generic IEEE 802.3u media interface,
>>>>>>> rev. 0: OUI
>>>>>>> 0x0001c1, model 0x0027
>>>>>>> octsctl0 at simplebus0: disabled
>>>>>>> octxctl0 at simplebus0: DWC3 rev 0x250a
>>>>>>> xhci0 at octxctl0, xHCI 1.0
>>>>>>> usb0 at xhci0: USB revision 3.0
>>>>>>> uhub0 at usb0 configuration 1 interface 0 "Generic xHCI root
>>>>>>> hub" rev
>>>>>>> 3.00/1.00 addr 1
>>>>>>> octxctl1 at simplebus0: DWC3 rev 0x250a
>>>>>>> xhci1 at octxctl1, xHCI 1.0
>>>>>>> usb1 at xhci1: USB revision 3.0
>>>>>>> uhub1 at usb1 configuration 1 interface 0 "Generic xHCI root
>>>>>>> hub" rev
>>>>>>> 3.00/1.00 addr 1
>>>>>>> "i2c" at simplebus0 not configured
>>>>>>> "i2c" at simplebus0 not configured
>>>>>>> com0 at simplebus0: ns16550a, 64 byte fifo
>>>>>>> com0: console
>>>>>>> com1 at simplebus0: ns16550a, 64 byte fifo
>>>>>>> com1: probed fifo depth: 0 bytes
>>>>>>> octmmc0 at simplebus0
>>>>>>> sdmmc0 at octmmc0: 8-bit, mmc high-speed
>>>>>>> "spi" at simplebus0 not configured
>>>>>>> "ocla0" at simplebus0 not configured
>>>>>>> "dma-engine" at simplebus0 not configured
>>>>>>> "dma-engine" at simplebus0 not configured
>>>>>>> octrng0 at iobus0 base 0x1400000000000 irq 0
>>>>>>> octpcie0 at iobus0: 3 ports
>>>>>>> octpcie0 port 0: link timeout
>>>>>>> octpcie0 port 1: reset timeout
>>>>>>> octpcie0 port 2: reset timeout
>>>>>>> scsibus0 at sdmmc0: 2 targets, initiator 0
>>>>>>> sd0 at scsibus0 targ 1 lun 0: <Kingston, MMC4GB, 0000> removable
>>>>>>> sd0: 3728MB, 512 bytes/sector, 7634944 sectors
>>>>>>> vscsi0 at root
>>>>>>> scsibus1 at vscsi0: 256 targets
>>>>>>> softraid0 at root
>>>>>>> scsibus2 at softraid0: 256 targets
>>>>>>> root on sd0a (285e489508f0f02b.a) swap on sd0b dump on sd0b
>>>>>>
>>>>>> can you try this?
>>>>>>
>>>>>> Index: uipc_socket.c
>>>>>> ===================================================================
>>>>>> RCS file: /cvs/src/sys/kern/uipc_socket.c,v
>>>>>> diff -u -p -r1.344 uipc_socket.c
>>>>>> --- uipc_socket.c 31 Oct 2024 12:51:55 -0000 1.344
>>>>>> +++ uipc_socket.c 1 Nov 2024 23:53:25 -0000
>>>>>> @@ -787,48 +787,36 @@ m_getuio(struct mbuf **mp, int atomic, l
>>>>>> {
>>>>>> struct mbuf *m, *top = NULL;
>>>>>> struct mbuf **nextp = ⊤
>>>>>> - u_long len, mlen;
>>>>>> - size_t resid = uio->uio_resid;
>>>>>> + u_long len, mlen, alen;
>>>>>> + int align = atomic ? roundup(max_hdr, sizeof(long)) : 0;
>>>>>> int error;
>>>>>> - do {
>>>>>> - if (top == NULL) {
>>>>>> - MGETHDR(m, M_WAIT, MT_DATA);
>>>>>> - mlen = MHLEN;
>>>>>> - } else {
>>>>>> - MGET(m, M_WAIT, MT_DATA);
>>>>>> - mlen = MLEN;
>>>>>> - }
>>>>>> + m = m_gethdr(M_WAIT, MT_DATA);
>>>>>> + mlen = MHLEN;
>>>>>> +
>>>>>> + for (;;) {
>>>>>> /* chain mbuf together */
>>>>>> *nextp = m;
>>>>>> nextp = &m->m_next;
>>>>>> - resid = ulmin(resid, space);
>>>>>> - if (resid >= MINCLSIZE) {
>>>>>> - MCLGETL(m, M_NOWAIT, ulmin(resid, MAXMCLBYTES));
>>>>>> - if ((m->m_flags & M_EXT) == 0)
>>>>>> + /* How much data we want to put in this mbuf? */
>>>>>> + len = ulmin(uio->uio_resid, space);
>>>>>> + /* How much space are we allocating for that data? */
>>>>>> + alen = align + len;
>>>>>> + if (alen > mlen) {
>>>>>> + MCLGETL(m, M_NOWAIT, ulmin(alen, MAXMCLBYTES));
>>>>>> + if (!ISSET(m->m_flags, M_EXT) && alen > MCLBYTES)
>>>>>> MCLGETL(m, M_NOWAIT, MCLBYTES);
>>>>>> - if ((m->m_flags & M_EXT) == 0)
>>>>>> - goto nopages;
>>>>>> - mlen = m->m_ext.ext_size;
>>>>>> - len = ulmin(mlen, resid);
>>>>>> - /*
>>>>>> - * For datagram protocols, leave room
>>>>>> - * for protocol headers in first mbuf.
>>>>>> - */
>>>>>> - if (atomic && m == top && len < mlen - max_hdr)
>>>>>> - m->m_data += max_hdr;
>>>>>> - } else {
>>>>>> -nopages:
>>>>>> - len = ulmin(mlen, resid);
>>>>>> - /*
>>>>>> - * For datagram protocols, leave room
>>>>>> - * for protocol headers in first mbuf.
>>>>>> - */
>>>>>> - if (atomic && m == top && len < mlen - max_hdr)
>>>>>> - m_align(m, len);
>>>>>> + if (ISSET(m->m_flags, M_EXT))
>>>>>> + mlen = m->m_ext.ext_size;
>>>>>> }
>>>>>> + /* Avoid pain from a stupid max_hdr value */
>>>>>> + if (align < mlen)
>>>>>> + m->m_data += align;
>>>>>> +
>>>>>> + /* How much data can we put in this mbuf? */
>>>>>> + len = ulmin(mlen, len);
>>>>>> error = uiomove(mtod(m, caddr_t), len, uio);
>>>>>> if (error) {
>>>>>> m_freem(top);
>>>>>> @@ -836,13 +824,19 @@ nopages:
>>>>>> }
>>>>>> /* adjust counters */
>>>>>> - resid = uio->uio_resid;
>>>>>> space -= len;
>>>>>> m->m_len = len;
>>>>>> top->m_pkthdr.len += len;
>>>>>> /* Is there more space and more data? */
>>>>>> - } while (space > 0 && resid > 0);
>>>>>> + if (space == 0 || uio->uio_resid == 0)
>>>>>> + break;
>>>>>> +
>>>>>> + align = 0;
>>>>>> +
>>>>>> + m = m_get(M_WAIT, MT_DATA);
>>>>>> + mlen = MLEN;
>>>>>> + }
>>>>>> *mp = top;
>>>>>> return 0;
>>>>>>
>>>>>
>>>>>
>>>
>>>
No comments:
Post a Comment