[openib-general] CM handling of REJ retry

[Sean Hefty]

Fab Tillier wrote:
> The IB spec, section 12.9.8.7 states that the local CID should be changed if
> retrying a REQ after receiving a REJ in the REQ Sent or REP Wait state.

What a nightmare.

> The current API seems to require uses to destroy their CID and allocate a
> new one before they can retry (i.e. it doesn't support REJ retry).  Do we
> want (and is it possible) to have the CM reserve the upper byte (or any
> other portion) of the LCID and use it as a revolving counter so users can
> just retry a request?  This would limit a system to 2^24 CIDs, which is
> probably more then enough.

The CM has complete control over the local comm id, so it can do 
whatever it wants to with it.  Cycling part of the comm id should work 
fine.

Also, given the current implementation, users need to re-create their 
comm id's with each connection in order to avoid using data that was 
stored with the previous connection.  (The fix is simple and involves 
clearing the stored data, but it's not there atm.)

- Sean