Re: How to parallelization

On 12/23/24 1:43 PM, Christian Schulte wrote:
> Not criticizing OpenBSD in any way. Let me try to explain a common use
> case. There is a data source capable of providing X bytes per second at
> max. The application needs to be setup in a way it can receive those X
> bytes per second without spin locking or waiting for data. If it would
> be "polling" too fast, it would slow down the whole system waiting for
> data. If it would be "polling" too slow, it would not be able to process
> those bytes fast enough. Those bytes need to be processed. So there is a
> receiving process which needs to be able to consume exactly those X
> bytes per second. That consumer also needs to be defined in a way it can
> process those bytes in parallel as fast as possible. Sizing the consumer
> too small, the producer will start spin locking or such and cannot keep
> up with the data rate it needs to process, because the consumer does not
> process the data fast enough. Sizing the consumer too big, the consumer
> will start spin locking or such waiting for the producer to provide more
> data. I am searching for an API to make the application adhere to those
> situations automatically. Data rate on the receiving part decreases,
> consumer part does not need to use Y processes in parallel all spin
> locking waiting for more data. Data rate on the receiving part
> increases, consumer needs to increase compute to not slow down the
> receiver. Does this make things more clear?
>
Thank you for your explanation.

I'm assuming that there multiple logical streams which are
extracted from the incoming packet streams and distributed
to multiple consumers.

Is it true that you are attempting to perfectly assign and utilize
all CPUs for packet & application processing?
If packet ordering is to be preserved it's not clear that
perfect allocation is possible.

>>>I am searching for an API to make the application adhere to those
>>>situations automatically

The only way I can see to achieve 100% perfect usage this is:
buffer the incoming packet stream deeply
inspect each packet to measure application resources needed
summarize those results over some time period
systemwide, measure resource capability and current utilization
determine systemwide resource allocation using some algorithm
adjust systemwide application resources
forward packets to each application

I worked on a network appliance which did complex resource
allocation while forwarding packets. It wasn't simple.

geoff steckel