[Users] interpreting ibdiagnet output

[Ira Weiny]

On Mon, 17 Sep 2012 20:36:03 -0500
John Valdes <valdes at anl.gov> wrote:

> On Mon, Sep 17, 2012 at 02:45:51PM -0700, Ira Weiny wrote:
> > Narayan Desai <narayan.desai at gmail.com> wrote:
> > 
> > > I found some reference to ipoib at speeds greater than 10 gbits
> > > requiring some manual setup in opensm; is this still the case?
> > 
> > Could you point me to the reference?  I just want to see what it is referring to.
> 
> Don't know if this is where Narayan saw it, but in the opensm(8)
> manpage, under the "PARTITION CONFIGURATION" section, it says:
> 
>        [...]
>        General file format:
> 
>        <Partition Definition>:<PortGUIDs list> ;
> 
>        Partition Definition:
> 
>        [PartitionName][=PKey][,flag[=value]][,defmember=full|limited]
> 
>        [...]
>        Currently recognized flags are:
> 
>         ipoib       - indicates that this partition may be used for IPoIB, as
>                       result IPoIB capable MC group will be created.
>         rate=<val>  - specifies rate for this IPoIB MC group
>                       (default is 3 (10GBps))
> 
> The fact that it says the default rate=3 (10GBps--I assume that's
> "bits") to me implies that IPoIB runs at 10 gbits unless configured
> otherwise in the opensm's partition.conf file, as Narayan says.  Is
> this true?  Or is it true for versions of OpenSM < N, for some value
> of N?  Is there any way (eg, some IB diag command to run), other than
> running a network benchmark, to determine the configured rate for
> IPoIB?

saquery can query the MCast groups.

saquery -g
saquery MCMR

These give slightly different output for historical reasons.

NOTE: "rate" returned is actually 2 fields (RateSelector (2bits) followed by Rate (6bits).  RateSelector is always 0x2 in a response.[$]

>From these groups you need to ID which groups are for IPoIB.  The IPoIB rfc defines ff12:401b:: and ff12:601b:: as the prefixes for link local groups for IPv4 and IPv6 respectively.[*]  IPv4 has a special case group for broadcast: ff12:401b:<P_key>::ffff:ffff

Ira

[*] http://tools.ietf.org/html/rfc4391#section-4
[$] IBTA Volume 1 v1.2.1 section 15.2.5.17

> 
> BTW, the manpage snippet above is from RHEL5's bundled opensm, which
> is version 3.3.3.
> 
> > The only place to preconfigure Mcast groups is in the "partitions.conf" file.  (I know, don't ask.)
> > 
> > I think you may want something like this line.
> > 
> > Default=0x7fff,ipoib,rate=7 : ALL = full;
> > 
> > This should set all the IPoIB Mcast groups to 40 Gb/sec.
> 
> That sounds right, or is at least consistent w/ the manpage.  In the
> absence of a partitions.conf file, tha manpage says opensm will create
> a single partition with:
> 
>   Default=0x7fff : ALL=full ;
> 
> so if the rate=3 if not specified, then I assume the default IPoIB MC
> group only runs at 10 Gb/s.
> 
> John
> _______________________________________________
> Users mailing list
> Users at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users


-- 
Ira Weiny
Member of Technical Staff
Lawrence Livermore National Lab
925-423-8008
weiny2 at llnl.gov