[ofa-general] CQEs for QP being destroyed/reset (was Re: [GIT PULL] please pull infiniband.git)

[Michael S. Tsirkin]

> Quoting Roland Dreier <rdreier at cisco.com>:
> Subject: Re: [ofa-general] Re: [GIT PULL] please pull infiniband.git
> 
>  > We have a rule that we must clean CQEs after QP is moved to RESET,
>  > do we not? How is it formulated?
> 
> Thanks for bringing that up...
> 
> I consider the CQ cleaning to be an internal mthca implementation
> detail, to avoid having to handle CQEs with a stale QPN after we free
> a QP.  The IB spec basically says that the status of completions for a
> QP is undefined:
> 
>     "The CI does not guarantee that CQEs generated for a QP prior to
>     its destruction can be retrieved from the CQ after that QP has
>     been destroyed."

This as I mentioned earlier simply means some CQEs might disappear
when you destroy the QP. And by the way, cleanup_cq in mthca
does exactly that - removes CQEs that were generated prior to
QP destruction.

> So now that you bring it up, I think it would be OK for a low-level
> driver to report completions for a QP even after the QP is destroyed.

Thanks for bringing this up, and reading this part of the spec I
found some interesting stuff.

Actually, the IB spec clarifies why there might not be any issues
and why the API we have might actually be a fine one: 10.2.4.4
                                                                           
	It is good programming practice to modify the QP into the Error state and
	retrieve the relevant CQEs prior to destroying the QP. Destroying a QP does
	not guarantee that CQEs of that QP are deallocated from the CQ upon
	destruction. Even if the CQEs are already on the CQ, it might not be 
	possible to retrieve them. It is good programming practice not to make any
	assumption on the number of CQEs in the CQ when destroying a QP. In order to
	avoid CQ overflow, it is recommended that all CQEs of the destroyed QP are
	retrieved from the CQ associated with it before resizing the CQ, attaching a
	new QP to the CQ or reopening the QP, if the CQ capacity is limited.

So it seems the ticket is that ULP must not destroy QP without
moving it to error and draining it first.

> And yes that does make putting the QP pointer in the ib_wc structure a
> mistake...

On the contrary, since ULPs must wait till all WRs complete, the API we have is
a fine one. But, IPoIB CM must be fixed to wait for all WRs to complete before
destroying the QP.

This is all dandy for send WRs. But I really have no idea how to interpret this
text in the spec, and what does "the relevant CQEs" mean in the context of a
receive queue connected to a SRQ.

Can someone comment on this last point?

-- 
MST