[openib-general] Some ib_mad.h Redirection Comments

[Sean Hefty]

On Thu, 05 Aug 2004 20:13:26 -0400
Hal Rosenstock <halr at voltaire.com> wrote:

> Are you talking about a client with multiple QPs and causing the
> requester to alternate amongst them via some (client) load balancing
> algorithm ? 

As an example:  I think that a CM on node 1 should be able to redirect connection request messages from node 2 to QP 23, and messages from node 3 to QP 34.  The CM should be responsible for allocating the QPs, sizing them appropriately, allocating the CQs, and controlling the processing that occurs on the redirected QPs/CQs.
 
> I don't think this by itself forces redirection above the API. I think
> it can still be handled transparently inside the GSI.

Trying to handle this "transparently" in the GSI puts too much policy in the GSI.  I think that it's necessary for the clients to manage their QPs, not the access layer.

> I think this depends on what the meaning of calling qp_redir is/should
> be. I think we still have differing assumptions about this.

I think that our assumptions differ.  My assumption is that qp_redir essentially just creates an ib_mad_agent structure, so that a call to ib_mad_post_send can post a work request to the proper QP.

> > The client would be responsible for caching the redirection.
> 
> This seems like a commonality which can be pushed down into the access
> layer and makes more sense when the access layer is handling the
> retransmissions on redirect. If it isn't, it makes less sense.

Sending to a redirected QP involves setting the remote_qpn, remote_qkey, and pkey_index in the ib_mad_send_wr to different values.  It's trivial for a client to set this.

> So back to that issue: if the access layer is already matching responses
> to requests (when timeout is enabled), I don't see why it doesn't handle
> the retransmission on receipt of redirect so each client doesn't need to
> implement this.

Currently, the access layer is not retransmitting MADs outside of RMPP.  When a response comes in, both the request and response may be handed to the user.  If the response is a redirect, the client can cache the redirection information and re-issue the request.  All future requests sent by that client can now automatically go to the correct location without the GSI having to snoop the destination, lookup whether that destination has been redirected, and modifying the outbound MAD.

Are you concerned about duplicating the management of a redirection table?  I think that redirection tables must be maintained per client.  Would adding support for redirection table management remove your concerns?