[openib-general] ib_mad: Scenarios for returning posted send MADs

[Hal Rosenstock]

There are two lists of posted send MADs: (1) a list of posted sends for
the port, and (2) another list per MAD agent. When a send is first
posted, it is placed on both lists until the send completion occurs and
then is removed from the port send list. The handling of the agent send
list is based on whether there is a timeout specified or not.

1. In the case that a client unregisters with the MAD layer, there is
code which cleans up the agent send list. However, it does not appear to
me that if the send completion occurs after the deregistration that this
completion is thrown away properly but rather a callback may be
performed. Did I miss something here ?

2. Another scenario for this is on WC errors which currently attempt to
restart the port. I am not sure all WC errors should do this. Perhaps
only IB_WC_FATAL_ERR and IB_WC_GENERAL_ERR. 

3. The final scenario is board (not currently possible) or module
removal. My concern here is about potential send callbacks (indicating
FLUSHED) to a potentially stale MAD agent. When the module is removed
non forceably, the clients (upper layer modules) would need to be
removed first, which should cause the proper deregistration (and these
MADs would be cancelled so there would be none to cleanup). I am not
sure what the rules for proper behavior are on forceable module removal.
Board removal would be similar to this (the forceable module removal
case).

-- Hal