[openib-general] [RFC] CM reject handling and redirection

[Sean Hefty]

I've started looking at the handling received rejects in the CM.  Two 
return codes of note:

* code 24 - Port and CM Redirection
* code 25 - Port Redirection

For the first case, it appears that the proper way to handle it is to 
simply expose the rejection to the user.  The user would need to issue 
a second REQ using a new path record.  Does this seem reasonable to 
everyone?

For the second case, the issue becomes more complex.  The CM is 
instructed to send to the port associated with the original REQ, but 
the connection is established to a separate port.

Currently, the CM sends to the port specified by the path record, 
ib_cm_req_param::primary_path, that will be used by the connection.  A 
port redirection REJ message changes this behavior.

After the REJ message is given to the user, a subsequent call to send a 
REQ will result in the CM sending the message to the new port.  For the 
CM to be able to send the REQ to the same port as the original REQ, it 
would need to store yet more state information regarding previous 
connection request attempts.  Since it's possible that the user could 
issue an unrelated REQ after receiving the REJ.

Should the CM try to maintain this state information, or should the API 
be expanded to allow the user to specify where to send the CM messages, 
or should we just ignore this reject code for now?   The needed state 
information would probably consist of: type of last message received, 
last REJ reason, last DGID, service ID of last REQ.  An expanded API 
could let the CM destination be optional, with a default to send based 
on the current path.

It's also not clear to me what effect this reject code has with respect 
to port failover.  Thoughts?

- Sean