[ewg] [PATCH] libibmad: Handle MAD redirection

[Hal Rosenstock]

On Tue, Jun 30, 2009 at 8:04 AM, Joachim Fenkes<fenkes at de.ibm.com> wrote:
> On Tuesday 30 June 2009 00:01, Hal Rosenstock wrote:
>> On Mon, Jun 29, 2009 at 8:10 AM, Joachim Fenkes<fenkes at de.ibm.com> wrote:
>
>> > Previously, libibmad reacted to GSI MAD responses with a "redirect" status
>> > by throwing an error. IBM eHCA adapters use redirection, so most
>> > infiniband_diags tools didn't work against eHCA.
>>
>> Are there GS classes other than PerfMgt which would be redirected by eHCA ?
>
> Not right now, no. If you're interested in the details of how and when the
> eHCA driver redirects, please have a look at drivers/infiniband/hw/ehca/ehca_sqp.c.

SL is always set to 0 and RespTimeValue is hardcoded.

>> > --- a/libibmad/src/gs.c
>> > +++ b/libibmad/src/gs.c
>> > @@ -70,7 +70,8 @@ uint8_t *pma_query_via(void *rcvbuf, ib_portid_t * dest, int port,
>> >        rpc.datasz = IB_PC_DATA_SZ;
>> >        rpc.dataoffs = IB_PC_DATA_OFFS;
>> >
>> > -       dest->qp = 1;
>> > +       if (!dest->qp)
>> > +               dest->qp = 1;
>>
>> Is this change part of this patch or unrelated/separate ?
>
> Part of the patch. Without this change, pma_query_via() would overwrite the
> redirected QP with QP1 again, and the MAD would never arrive at the right
> destination.
>
>> > +               /* check for exact match instead of only the redirect bit;
>> > +                * that way, weird statuses cause an error, too */
>> > +               if (status == IB_MAD_STS_REDIRECT) {
>> > +                       /* update dport for next request and retry */
>> > +                       dport->lid = mad_get_field(mad, 64, IB_CPI_REDIRECT_LID_F);
>> > +                       dport->qp = mad_get_field(mad, 64, IB_CPI_REDIRECT_QP_F);
>> > +                       dport->qkey = mad_get_field(mad, 64, IB_CPI_REDIRECT_QKEY_F);
>>
>> Are those the only 3 fields which eHCA changes on a redirect ? There
>> may be others we would want to add in here (PKey, SL, ...) ?
>
> Yeah, I agree on the SL, I can add it to the patch.
>
> At first, I also tried to set the PKey, but ClassPortInfo specifies a PKey
> while ib_portid_t needs a PKey Index, and I found no way of converting
> between the two,

It's available via libibumad. Note that umad version 5 is needed for
pkey index support.

> so I left it at zero. Incidentally, there isn't a single
> code line in management.git that actually changes the pkey_index from its
> init value of 0, so I figured that omission couldn't be too bad.

Agreed. I think you're referring to infiniband-diags and not opensm.

> Then there's the GRH stuff, but I refrained from coding that because I
> wouldn't be able to test it

That's fine for now IMO. I think there's only some minimal GRH support
now elsewhere in the diags and this needs more work in general.

> -- InfiniBand isn't going to evolve beyond a
> single subnet any time toon, is it?

There's ongoing work in this area but not sure how soon is soon...

>> Also, are the offsets above correct ?
>
> Yes, they are, I tested. The ClassPortInfo data starts at offset 64 in the
> MAD, and I didn't find a constant for this in mad.h.
>
>> Depending on which GS classes are to be supported for redirection, we
>> may want to do something similar to the rmpp equivalent of this
>> routine too.
>
> The spec says in 13.5.2 that "The SA as well as each GSA may individually
> support this mechanism or not", so we should probably be prepared for any GS
> class to redirect. I don't care much about RMPP, though, so I left it alone.

Understood.

-- Hal

> Regards,
>  Joachim
>
>
>
>