[openib-general] [PATCH/RFC] IB/ipoib: add selective tx signaling

[Michael S. Tsirkin]

> Quoting Roland Dreier <rdreier at cisco.com>:
> Subject: Re: [PATCH/RFC] IB/ipoib: add selective tx signaling
> 
> I played with this idea too a long time ago, but I don't think this
> sort of naive implementation is acceptable.  Having packets that take
> a potentially unbounded amount of time before the send is completed
> and the skb is freed just causes too many problems -- for example the
> ping failures that you noted.
> 
> Unless you can come up with a way that makes sure that all skbs are
> completed even in low-traffic situations, I don't think this is
> mergeable -- it's just too much of a usability nightmare to have a
> flag that is essentially "break some workloads in a mysterious way to
> make some benchmarks run a little faster."

Yea, I really think the selective signalling is only good for cases where
you post several WRs in one go, anything else is a misuse.

Thinking about this more - why does this patch help some benchmarks?
The amount of work it takes for the hardware to generate a completion
is likely negligeable, and we still are scanning the same amount
of TX WRs in a loop to unmap/free them.

If you think about it this way, it becomes clear that your workload,
for some reason, hits a path where you get an event very fast
after the first completion and there is only a small number of completions
to handle. So your patch helps just by delaying the event handler until
there's more work to do. And I expect it wouldn't help TCP much if at all
as there are RX WRs per each couple of TX WRs.

So, can't we do attack the problem directly?
I am thinking about detecting that we are getting a completion each
couple of microseconds, and just polling a bit more (even if CQ is empty)
simply to avoid paying 10s micro for the next interrupt.
This is just a fuzzy idea, but maybe this can work?


-- 
MST