[openib-general] Re: uverbs performance; ibv_pingpong poll vs sleep

[Michael S. Tsirkin]

Quoting r. Steven Wooding <sinate at yahoo.com>:
> Subject: uverbs performance; ibv_pingpong poll vs sleep
> 
> Hi,
>  
> I wonder if anyone could help me with uverbs performance.
>  
> I have compaired the data rates using ibv_pingpong with and without the -e
> option. Therefore, using polling and waiting the CQ events (sleeping).
>  
> For a data message size of 16K I get the following results:
>  
> Poll: 718 MB/s with CPU at 100%
> Sleep: 479 MB/s with CPU at 16%

What do you do for CPU utilisation measurement?

> I suppose the decrease in throughput is due to the time it takes to get the CQ
> event.

Yes, pingpong is not really a bandwidth benchmark.
It never has more than one outstanding transaction on a qp,
so its bandwidth is very sensitive to roundtrip times.

> Is there any way to trade off the data rate with the CPU usage (I was thinking
> of some timeout from polling).
>  
> Any suggests would be very welcome.
>  
> Regards,
>  
> Steve.

Steve, I think an application should create some kind of
pipelining, performing multiple sends before waiting for
events.

I have a working example of how this can be done, but unfortunately
its not yet ready for publication.

The idea basically is to have n outstanding buffers and tx of size n.
Count the number of send requests sent and completed.
Now basically replace each post send with

	while (send - completed < n)
		post send


As n grows, you shall see different cpu utilization/bandwidth points.


-- 
MST