[openib-general] [PATCH] cancel outstanding MADs when deregistering

[Sean Hefty]

On Tue, 28 Sep 2004 12:50:52 -0700
Roland Dreier <roland at topspin.com> wrote:

> It looks OK for current functionality but I think it will have to
> change to support cancelling sends.  (Cancelling sends is required
> for consumers that start a query with a long timeout and then want to
> unload or something like that).

I'm aware that there's currently no functionality for canceling a single MAD.  This would require a new API, and possibly changes to the send MAD API.
 
> When someone asks to cancel a send you have to tell the consumer
> whether the send was canceled or had already finished (so that they
> know whether the resources have already been freed).  You have to make
> sure that you don't say that the send wasn't canceled and then return
> with the send completion handler still running on some other CPU,
> because then the consumer will probably corrupt context.

I am aware of this issue.  My plan to handle it was to search for MADs to cancel, in case the cancel call came after the MAD had already been returned to the user.

> That means that you can't just remove things from your pending list
> like the code does now -- you have to leave them there and mark them
> "callback running" or something like that.  It ends up being even more
> complicated unfortunately.

The cancel code that's there should work if deregistration is called at the same time that a send operation completes.  A reference it taken on the send MAD work request structure while the work request is posted to the QP.  A second reference is taken if the MAD has a timeout, meaning that a response is expected.  The current cancel code (called only during deregistration) only releases the reference taken for MADs that have a timeout.  If no other references remain on the send MAD work request, it is canceled immediately.  Otherwise, it is marked as flushed, and will be completed once other references go away.  (Other references would be for multiple work requests posted to the QP for RMPP.)

As a last note, my intention is to have the code callback the user for every MAD that they've sent.