On Mon, Sep 17, 2012 at 02:45:51PM -0700, Ira Weiny wrote:
> Narayan Desai <narayan.desai at gmail.com> wrote:
>
> > I found some reference to ipoib at speeds greater than 10 gbits
> > requiring some manual setup in opensm; is this still the case?
>
> Could you point me to the reference? I just want to see what it is referring to.
Don't know if this is where Narayan saw it, but in the opensm(8)
manpage, under the "PARTITION CONFIGURATION" section, it says:
[...]
General file format:
<Partition Definition>:<PortGUIDs list> ;
Partition Definition:
[PartitionName][=PKey][,flag[=value]][,defmember=full|limited]
[...]
Currently recognized flags are:
ipoib - indicates that this partition may be used for IPoIB, as
result IPoIB capable MC group will be created.
rate=<val> - specifies rate for this IPoIB MC group
(default is 3 (10GBps))
The fact that it says the default rate=3 (10GBps--I assume that's
"bits") to me implies that IPoIB runs at 10 gbits unless configured
otherwise in the opensm's partition.conf file, as Narayan says. Is
this true? Or is it true for versions of OpenSM < N, for some value
of N? Is there any way (eg, some IB diag command to run), other than
running a network benchmark, to determine the configured rate for
IPoIB?
BTW, the manpage snippet above is from RHEL5's bundled opensm, which
is version 3.3.3.
> The only place to preconfigure Mcast groups is in the "partitions.conf" file. (I know, don't ask.)
>
> I think you may want something like this line.
>
> Default=0x7fff,ipoib,rate=7 : ALL = full;
>
> This should set all the IPoIB Mcast groups to 40 Gb/sec.
That sounds right, or is at least consistent w/ the manpage. In the
absence of a partitions.conf file, tha manpage says opensm will create
a single partition with:
Default=0x7fff : ALL=full ;
so if the rate=3 if not specified, then I assume the default IPoIB MC
group only runs at 10 Gb/s.
John