[openib-general] Completion callback /teardown race

[Fabian Tillier]

On 9/20/06, Michael S. Tsirkin <mst at mellanox.co.il> wrote:
> Quoting r. Fabian Tillier <ftillier at silverstorm.com>:
> > > There's a mistake here. Mellanox HCAs will generate an event upon
> > > ib_req_notify_cq only if new completions has arrived after the previous event
> > > has been reported.
> >
> > Thanks for correcting me - I expected my memory to be a bit rusty.  In
> > this case, is there any benefit in polling before calling
> > ib_req_notify_cq?
> >
> > > AFAIK this is IBTA spec compliant.
> >
> > Yes, I believe it is too.  Do you know if there is any impact on
> > performance in doing the following for completion processing:
> >
> > ib_req_notify_cq
> > poll_cq until empty
>
> Some additional polling has a chance to improve performance on any
> hardware: it increases the chance that you do a cheap poll for completion
> instead of getting a (typically expensive) notification interrupt.
> And its a win on any hardware to delay ib_req_notify_cq
> as long as possible, so that a single event reports as many completions
> as possible.

Ok, now you have me confused.  Based on what you said for Mellanox
HCAs, a new CQ event will be generated when the CQ is rearmed if any
CQEs where written since the last event was generated.  To me this
means that it doesn't matter if these CQEs where reaped or not.

That is, at t0 you have a CQE written and a CQ notification.  At t1
you have nother CQE written.  At t2 you poll both CQEs and rearm.
Since the CQE from t1 was written after the last event, I would expect
(based on your description) that I would get another CQ notification,
eventhough I already reaped the CQE.

Did you mean that the hardware will only generate a new event if there
are any un-reaped CQEs?

- Fab