[ofa-general] Re: [ewg] [PATCH] libibmad: Handle MAD redirection

[Joachim Fenkes]

On Tuesday 30 June 2009 00:01, Hal Rosenstock wrote:
> On Mon, Jun 29, 2009 at 8:10 AM, Joachim Fenkes<fenkes at de.ibm.com> wrote:

> > Previously, libibmad reacted to GSI MAD responses with a "redirect" status
> > by throwing an error. IBM eHCA adapters use redirection, so most
> > infiniband_diags tools didn't work against eHCA.
> 
> Are there GS classes other than PerfMgt which would be redirected by eHCA ?

Not right now, no. If you're interested in the details of how and when the
eHCA driver redirects, please have a look at drivers/infiniband/hw/ehca/ehca_sqp.c.
 
> > --- a/libibmad/src/gs.c
> > +++ b/libibmad/src/gs.c
> > @@ -70,7 +70,8 @@ uint8_t *pma_query_via(void *rcvbuf, ib_portid_t * dest, int port,
> >        rpc.datasz = IB_PC_DATA_SZ;
> >        rpc.dataoffs = IB_PC_DATA_OFFS;
> >
> > -       dest->qp = 1;
> > +       if (!dest->qp)
> > +               dest->qp = 1;
> 
> Is this change part of this patch or unrelated/separate ?

Part of the patch. Without this change, pma_query_via() would overwrite the
redirected QP with QP1 again, and the MAD would never arrive at the right
destination.
 
> > +               /* check for exact match instead of only the redirect bit;
> > +                * that way, weird statuses cause an error, too */
> > +               if (status == IB_MAD_STS_REDIRECT) {
> > +                       /* update dport for next request and retry */
> > +                       dport->lid = mad_get_field(mad, 64, IB_CPI_REDIRECT_LID_F);
> > +                       dport->qp = mad_get_field(mad, 64, IB_CPI_REDIRECT_QP_F);
> > +                       dport->qkey = mad_get_field(mad, 64, IB_CPI_REDIRECT_QKEY_F);
> 
> Are those the only 3 fields which eHCA changes on a redirect ? There
> may be others we would want to add in here (PKey, SL, ...) ?

Yeah, I agree on the SL, I can add it to the patch.

At first, I also tried to set the PKey, but ClassPortInfo specifies a PKey
while ib_portid_t needs a PKey Index, and I found no way of converting
between the two, so I left it at zero. Incidentally, there isn't a single
code line in management.git that actually changes the pkey_index from its
init value of 0, so I figured that omission couldn't be too bad.

Then there's the GRH stuff, but I refrained from coding that because I
wouldn't be able to test it -- InfiniBand isn't going to evolve beyond a
single subnet any time toon, is it?

> Also, are the offsets above correct ?

Yes, they are, I tested. The ClassPortInfo data starts at offset 64 in the
MAD, and I didn't find a constant for this in mad.h.
 
> Depending on which GS classes are to be supported for redirection, we
> may want to do something similar to the rmpp equivalent of this
> routine too.

The spec says in 13.5.2 that "The SA as well as each GSA may individually
support this mechanism or not", so we should probably be prepared for any GS
class to redirect. I don't care much about RMPP, though, so I left it alone.

Regards,
  Joachim