[openib-general] [RFC] send side QP redirection

[Hal Rosenstock]

On Sat, 2005-09-17 at 14:58, Sean Hefty wrote:
> >> struct ib_mad_av {
> >> 	struct ib_ah *ah;
> >> 	u32 remote_qpn;
> >> 	u32 remote_qkey;
> >> 	u16 pkey_index;
> >> };
> >
> >What about SL and the other redirect GRH fields (TC and FL) ?
> 
> These would have been specified through the ib_ah_attr when the destination was
> added.  I think that these are the four fields needed to allocate and send a
> MAD.  To clarify, an ib_mad_av specifies a <mgmt_class, lid, gid> tuple, which I
> refer to as a destination.

Aren't LID and GID also part of ib_ah_attr struct too ?

> >> struct ib_mad_av* ib_insert_mad_dest(struct ib_mad_agent*,
> >> 				     struct ib_ah_attr*,
> >> 				     mgmt_class, lid, gid, pkey);
> >
> >So the last three parameters are transferred from the received
> >ClassPortInfo.
> 
> Not exactly, the destination is inserted based on mgmt_class, lid, and gid
> before sending any request.  These are likely coming from a path record.  The
> pkey is translated into an index and returned as part of the MAD address vector.
> Basically, a user would call this routine in place of ib_create_ah.

But this call takes an ib_ah_attr struct so I don't see how this call is
in place of ib_create_ah.

> >> struct ib_mad_av* ib_get_mad_av(struct ib_mad_agent*,
> >> 				mgmt_class, lid, gid);
> >
> >I'm missing why LID and GID are needed here and why pkey is missing.
> >Is there more than 1 struct ib_mad_av allowed per mgmt_class ?
> 
> There will be one ib_mad_av per mgmt_class per remote destination.  A remote
> destination is identified by the class and LID.  (For the initial
> implementation, the GID will essentially be ignored.)

Got it.

> >> /*
> >>   * TBD: need to determine when to remove a destination.  Can remove
> >>   * always if references go to 0.  Can add a delay before removal.  Can
> >>   * maintain destinations that have been redirected. ?
> >>   */
> >> void ib_free_mad_av(struct ib_mad_av*);
> >>
> >> ib_redirect_mads(struct ib_mad_av*, struct ib_mad_recv_wc*,
> >> 		 struct ib_class_port_info*);

Does this return void or int ?

> >> /* GID redirection with unknown LID would be deferred */
> >> int ib_get_mad_redirect_path(...);
> >
> >Just curious why ?
> 
> The spec permits redirection to another GID without specifying the LID.  This
> requires that the requester send a path record query to the SA to obtain the
> rest of the information before it can send to the redirected QP.  This results
> in an asynchronous operation, which becomes more difficult to deal with.  The
> original MAD and others destined for that QP now need to be queued until the
> query completes and the full destination is known.

I can see why this functionality should be deferred from an complexity
standpoint. Any idea where this might be used rather than LID
redirection ?

> >> Questions or thoughts?
> >Is multiple redirection handled by this ?
> 
> It should be.  ib_redirect_mads uses the existing ib_mad_av to record that
> redirection has occurred.  This should allow for multiple redirection.

Understood. If a multiple redirection occurred, this would be called
multiple times.

> >Would these semantics need to be extended to user space by user_mad ?
> 
> I can't think of a reason why they couldn't.  It would only be called by
> userspace clients that initiate sending MADs.

That would be the next step.

-- Hal