Dear Andrew,
Thank you very much for all your advice.
> Unattended upgrades via sysupgrade usually go smoothly, but if not,
> you'll have the equivalent of physical access.
It was smooth indeed.
> Your paper looks interesting; it will be informative to see how much
> OpenBSD has improved post-7.1 (and -current may have even further
> improvements than 7.2, but I'll defer to others on those details).
>
> -Andrew
I did an IPv6 packet forwarding performance measurement. The results of
the 10 tests had some fluctuations. Here are the raw results (time is in
JST, the last column shows the rate):
root@p094:~/siitperf/results# cat
p094-10G-p095-OBSD72-MP-RFC4814-IPv6-b-84-2-2-60-2000-200-2023-03-04-0326/rate.csv
No, Size, Dir, n, m, Duration, Initial Rate, Timeout, Error, Date,
Iterations needed, rate
1, 84, b, 2, 2, 60, 200000, 2000, 200, 2023-03-04 00:17:08, 11, 279101
2, 84, b, 2, 2, 60, 200000, 2000, 200, 2023-03-04 00:38:06, 11, 279491
3, 84, b, 2, 2, 60, 200000, 2000, 200, 2023-03-04 00:59:03, 11, 266991
4, 84, b, 2, 2, 60, 200000, 2000, 200, 2023-03-04 01:20:03, 11, 274804
5, 84, b, 2, 2, 60, 200000, 2000, 200, 2023-03-04 01:41:00, 11, 262695
6, 84, b, 2, 2, 60, 200000, 2000, 200, 2023-03-04 02:02:00, 11, 282616
7, 84, b, 2, 2, 60, 200000, 2000, 200, 2023-03-04 02:22:57, 11, 281445
8, 84, b, 2, 2, 60, 200000, 2000, 200, 2023-03-04 02:43:53, 11, 278710
9, 84, b, 2, 2, 60, 200000, 2000, 200, 2023-03-04 03:04:51, 11, 275195
10, 84, b, 2, 2, 60, 200000, 2000, 200, 2023-03-04 03:25:50, 11, 241210
root@p094:~/siitperf/results#
My earlier results with OpenBSD 7.1 were much more consistent.
I did two tests with 300,000 fps and and one with 400,000 fps (per
direction) to check what top shows:
load averages: 0.45, 0.41, 0.20 p095.lab.starbed.org 05:04:11
36 processes: 35 idle, 1 on processor up 0 days 00:03:19
CPU00 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU01 states: 0.0% user, 0.0% nice, 81.0% sys, 11.2% spin, 0.0% intr,
7.8% idle
CPU02 states: 0.2% user, 0.0% nice, 66.9% sys, 5.6% spin, 0.0% intr,
27.3% idle
CPU03 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU04 states: 0.0% user, 0.0% nice, 20.6% sys, 1.0% spin, 0.0% intr,
78.4% idle
CPU05 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU06 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU07 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU08 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU09 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 34.3% intr,
65.7% idle
CPU10 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU11 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU12 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU13 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU14 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU15 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU16 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU17 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU18 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU19 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU20 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU21 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU22 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU23 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU24 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU25 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 30.3% intr,
69.7% idle
CPU26 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU27 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU28 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU29 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU30 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU31 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
Memory: Real: 33M/1438M act/tot Free: 371G Cache: 715M Swap: 0K/256M
load averages: 0.59, 0.54, 0.29 p095.lab.starbed.org 05:06:56
36 processes: 35 idle, 1 on processor up 0 days 00:06:05
CPU00 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU01 states: 0.0% user, 0.0% nice, 80.0% sys, 11.4% spin, 0.0% intr,
8.6% idle
CPU02 states: 0.0% user, 0.0% nice, 85.8% sys, 5.8% spin, 0.0% intr,
8.4% idle
CPU03 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU04 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU05 states: 0.0% user, 0.0% nice, 0.2% sys, 0.0% spin, 0.0% intr,
99.8% idle
CPU06 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU07 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU08 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU09 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 30.7% intr,
69.3% idle
CPU10 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU11 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU12 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU13 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU14 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU15 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU16 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU17 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU18 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU19 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU20 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU21 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU22 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU23 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU24 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU25 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 31.5% intr,
68.5% idle
CPU26 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU27 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU28 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU29 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU30 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU31 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
Memory: Real: 33M/1437M act/tot Free: 371G Cache: 715M Swap: 0K/256M
load averages: 0.92, 0.68, 0.39 p095.lab.starbed.org 05:09:34
36 processes: 35 idle, 1 on processor up 0 days 00:08:43
CPU00 states: 0.0% user, 0.0% nice, 10.2% sys, 0.8% spin, 0.0% intr,
89.0% idle
CPU01 states: 0.0% user, 0.0% nice, 57.3% sys, 6.8% spin, 0.0% intr,
35.9% idle
CPU02 states: 0.0% user, 0.0% nice, 39.5% sys, 3.6% spin, 0.0% intr,
56.9% idle
CPU03 states: 0.0% user, 0.0% nice, 11.6% sys, 0.2% spin, 0.0% intr,
88.2% idle
CPU04 states: 0.0% user, 0.0% nice, 44.7% sys, 3.8% spin, 0.0% intr,
51.5% idle
CPU05 states: 0.0% user, 0.0% nice, 5.2% sys, 0.4% spin, 0.0% intr,
94.4% idle
CPU06 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU07 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU08 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU09 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 38.5% intr,
61.5% idle
CPU10 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU11 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU12 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU13 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU14 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU15 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU16 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU17 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU18 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU19 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU20 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU21 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU22 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU23 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU24 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU25 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 37.7% intr,
62.3% idle
CPU26 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU27 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU28 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU29 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU30 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
CPU31 states: 0.0% user, 0.0% nice, 0.0% sys, 0.0% spin, 0.0% intr,
100% idle
Memory: Real: 33M/1443M act/tot Free: 371G Cache: 715M Swap: 0K/256M
All interrupts were processed by same two cores (CPU09 and CPU25). It is
so because I used a single IPv6 address pair with bidirectional traffic
thus the interrupts were hashed to two cores.
It seems that packet forwarding were executed by 2-5 cores: only they
had more than 1% system load. It is definitely and improvement compared
to OpenBSD 7.1, where only a single core did all that work.
As for the performance, the above results are frames per second per
direction values, so (except for the last test) they mean more than
500,000fps were forwarded. The median IPv6 forwarding performance of
OpenBSD 7.1 was 331,054fps, so it is a significant improvement.
And I suspect that maybe the root cause of the fluctuations could be an
"unfortunate" scheduling of multiple packet forwarding tasks to the same
core.
I hope that perhaps my results could help OpenBSD developers to improve
the performance further.
Best regards,
Gábor
No comments:
Post a Comment