[Users] Weird IPoIB issue

[Hal Rosenstock]

Xsigo's SM is not "straight" OpenSM. They have some proprietary
enhancements and it may be based on old vintage of OpenSM. You will likely
need to work with them/Oracle now on issues.

Lack of a partitions file does mean default partition and default rate (10
Gbps) so from what I saw all ports had sufficient rate to join MC group.

There are certain topology requirements for running various routing
algorithms. Did you try updn or dnup ?

The key is determining whether the IPoIB broadcast group is setup
correctly. What MLID is the group built on (usually 0xc000) ? What are the
group parameters (rate, MTU) ? Are all members that are running IPoIB
joined ? Is the group routed to all such members ? There are
infiniband-diags for all of this.


On Mon, Oct 28, 2013 at 12:19 PM, Robert LeBlanc <robert_leblanc at byu.edu>wrote:

> OpenSM (the SM runs on Xsigo so they manage it) is using minhop. I've
> loaded the ibnetdiscover output into ibsim and run all the different
> routing algorithms against it with and without scatter ports. Minhop had
> 50% of our hosts running all paths through a single IS5030 switch (at least
> the LIDs we need which represent Ethernet and Fibre Channel cards the hosts
> should communicate with). Ftree, dor, and dfsssp failed back to minhop, the
> others routed more paths through the same IS5030 in some cases increasing
> our host count with single point of failure to 75%.
>
> As far as I can tell there is no partitions.conf file so I assume we are
> using the default partition. There is an opensm.opts file, but it only
> specifies logging information.
> # SA database file name
> sa_db_file /var/log/opensm-sa.dump
>
> # If TRUE causes OpenSM to dump SA database at the end of
> # every light sweep, regardless of the verbosity level
> sa_db_dump TRUE
>
> # The directory to hold the file OpenSM dumps
> dump_files_dir /var/log/
>
> The SM node is:
> xsigoa:/opt/xsigo/xsigos/current/ofed/etc# ibaddr
> GID fe80::13:9702:100:979 LID start 0x1 end 0x1
>
> We do have Switch-X in two of the Dell m1000e chassis but the cards, ports
> 17-32, are FDR10 (the switch may be straight FDR, but I'm not 100% sure).
> The IS5030 are QDR which the Switch-X are connected to, the switches in the
> Xsigo directors are QDR, but the Ethernet and Fibre Channel cards are DDR.
> The DDR cards will not be running IPoIB (at least to my knowledge they
> don't have the ability), only the hosts should be leveraging IPoIB. I hope
> that clears up some of your questions. If you have more, I will try to
> answer them.
>
>
>
>
> Robert LeBlanc
> OIT Infrastructure & Virtualization Engineer
> Brigham Young University
>
>
> On Mon, Oct 28, 2013 at 9:57 AM, Hal Rosenstock <hal.rosenstock at gmail.com>wrote:
>
>> What routing algorithm is configured in OpenSM ? What does your
>> partitions.conf file look like ? Which node is your OpenSM ?
>>
>> Also, I only see QDR and DDR links although you have Switch-X so I assume
>> all FDR ports are connected to slower (QDR) devices. I don't see any FDR-10
>> ports but maybe they're also connected to QDR ports so show up as QDR in
>> the topology.
>>
>> There are DDR CAs in Xsigo box but not sure whether or not they run IPoIB.
>>
>> -- Hal
>>
>>
>> On Sun, Oct 27, 2013 at 9:46 PM, Robert LeBlanc <robert_leblanc at byu.edu>wrote:
>>
>>> Since you guys are amazingly helpful, I thought I would pick your brains
>>> in a new problem.
>>>
>>> We have two Xsigo directors cross connected to four Mellanox IS5030
>>> switches. Connected to those we have four Dell m1000e chassis each with two
>>> IB switches (two chassis have QDR and two have FDR10). We have 9 dual-port
>>> rack servers connected to the IS5030 switches. For testing purposes we have
>>> an additional Dell m1000e QDR chassis connected to one Xsigo director and
>>> two dual-port FDR10 rack servers connected to the other Xsigo director.
>>>
>>> I can get IPoIB to work between the two test rack servers connected to
>>> the one Xsigo director. But I can not get IPoIB to work between any blades
>>> either right next to each other or to the working rack servers. I'm using
>>> the same exact live CentOS ISO on all four servers. I've checked opensm and
>>> the blades have joined the multicast group 0xc000 properly. tcpdump
>>> basically says that traffic is not leaving the blades. tcpdump also shows
>>> no traffic entering the blades from the rack servers. An ibtracert using
>>> 0xc000 mlid shows that routing exists between hosts.
>>>
>>> I've read about MulticastFDBTop=0xBFFF but I don't know how to set it
>>> and I doubt it would have been set by default.
>>>
>>> Anyone have some ideas on troubleshooting steps to try? I think Google
>>> is tired of me asking questions about it.
>>>
>>> Thanks,
>>>
>>> Robert LeBlanc
>>> OIT Infrastructure & Virtualization Engineer
>>> Brigham Young University
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at lists.openfabrics.org
>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/users/attachments/20131028/58bf6a77/attachment.html>