[openib-general] smpdump and current MAD layer

[Sean Hefty]

Hal Rosenstock wrote:
>>I'd like to place the snooping code in as few places as possible, but 
>>still be able to snoop locally processed MADs.  Ideally a MAD should be 
>>snooped exactly once, which requires some extra care when handling QP 
>>errors.  Snooping in the completion handling allows the MAD layer to 
>>own the thread that performs the callback.  Calling clients back on the 
>>outbound path puts the callback in an arbitrary thread context.
> 
> While there are arguments for snooping when the MAD is posted to the QP,
> I think it makes more sense to snoop it when it completes. In addition
> to the thread reason you cite, one could also pass the completion status
> which would give more information on what really happened (which might
> be useful). I don't see a need to instrument this more than once on the
> send side either.

It would be difficult to snoop all MADs without snooping in multiple 
locations.  For example, local MADs do not generate CQ entries, and are 
  completed from the sender's thread.  (I think we may want to change 
the threading at some point, so it may not matter long term.)  It's 
also not clear if a snooper would want to know if a request received a 
reply, snoop separate send completions in the case of RMPP, or snoop 
RMPP ACKs.  (For RMPP debugging, I want to snoop at the lowest level.)

Right now, I've tried to design the API/code to be fairly flexible 
about where snooping can occur (since I'm not sure where the best place 
to do this is), but will likely only snoop in one location (after 
completions).

- Sean