[openib-general] Re: ib_mad: Scenarios for returning posted send MADs

[Sean Hefty]

On Tue, 05 Oct 2004 09:31:43 -0400
Hal Rosenstock <halr at voltaire.com> wrote:

> Do you mean that you should never get a callback for a mad_send_wr if
> (rather than unless) it's reference count is at least one ?

There will never be a completion callback associated with a mad_send_wr unless its reference count is >= 1.
 
> Cancelling the MAD decrements the reference count which has the effect
> of moving the callback one stage ahead. Either the send completion has
> already occurred in which case the reference count will be 0 or negative
> and the callback will be invoked immediately or it has not yet occurred
> so is invoked when the send completion occurs subsequently. The latter
> case leaves the MAD on both the port and agent send lists until it
> occurs. There are the other removal scenarios which interplay with this.

The reference count for a canceled MAD is only decremented *if there is a timeout specified*.  The existence of a timeout value indicates that a response is expected.  A mad_send_wr with a timeout has a reference count of 2 initially.  It is decremented by one when the send completion callback is invoked.  It is decremented if a response matches the request.  In order for the request to match, it must still have a timeout value.

If a MAD is canceled, its reference count is decremented if a timeout is given.  The timeout is set to 0 in this case to prevent responses from matching with the MAD.

 
> In any case, what should the MAD layer do when there are posted sends on
> an agent list ? Should it just dump them and not attempt to make a
> callback ? The bad side of this is that there are legitimate scenarios
> where this might occur. Not making the send callback has a number of
> side effects beyond the individual client (in terms of PCI mapping and
> memory leakage).

Currently, the MAD layer cancels all outstanding sends when the client deregisters, then waits for those sends to complete.  The client receives a callback for all sends that it has posted, including those posted when it calls deregister.