[openib-general] Completion callback /teardown race

[Michael S. Tsirkin]

Quoting r. Fabian Tillier <ftillier at silverstorm.com>:
> > There's a mistake here. Mellanox HCAs will generate an event upon
> > ib_req_notify_cq only if new completions has arrived after the previous event
> > has been reported.
> 
> Thanks for correcting me - I expected my memory to be a bit rusty.  In
> this case, is there any benefit in polling before calling
> ib_req_notify_cq?
> 
> > AFAIK this is IBTA spec compliant.
> 
> Yes, I believe it is too.  Do you know if there is any impact on
> performance in doing the following for completion processing:
> 
> ib_req_notify_cq
> poll_cq until empty

Some additional polling has a chance to improve performance on any
hardware: it increases the chance that you do a cheap poll for completion
instead of getting a (typically expensive) notification interrupt.
And its a win on any hardware to delay ib_req_notify_cq
as long as possible, so that a single event reports as many completions
as possible.

That's why it's common to e.g.

poll_cq until empty
ib_req_notify_cq
poll_cq until empty

this might work well for bursty traffic, where once CQ is empty it will stay
empty for a while.

There's no reason why polling twice will work best in all cases however -
it's easy to invent other heuristics:

for(i=0;i<1000;++i)
	poll_cq until empty
ib_req_notify_cq
for(i=0;i<10;++i)
	poll_cq until empty

etc.

what works best depends on the application.

-- 
MST