[Users] Weird IPoIB issue

Mon Oct 28 10:40:34 PDT 2013

saquery -g should show what MGID is mapped to MLID 0xc000 and the group
parameters.

When you say 10 Gbps max, is that multicast or unicast ? That limit is only
on the multicast.

On Mon, Oct 28, 2013 at 1:28 PM, Robert LeBlanc <robert_leblanc at byu.edu>wrote:

> Well, that can explain why I'm only able to get 10 Gbps max from the two
> hosts that are working.
>
> I have tried updn and dnup and they didn't help either. I think the only
> thing that will help is Automatic Path Migration is it tries very hard to
> route the alternative LIDs through different systemguids. I suspect it
> would require re-LIDing everything which would mean an outage. I'm still
> trying to get answers from Oracle if that is even a possibility. I've tried
> seeding some of the algorithms with information like root nodes, etc, but
> none of them worked better.
>
> The MLID 0xc000 exists and I can see all the nodes joined to the group
> using saquery. I've checked the route using ibtracert specifying the MLID.
> The only thing I'm not sure how to check is the group parameters. What tool
> would I use for that?
>
> Thanks,
>
>
> Robert LeBlanc
> OIT Infrastructure & Virtualization Engineer
> Brigham Young University
>
>
> On Mon, Oct 28, 2013 at 11:16 AM, Hal Rosenstock <hal.rosenstock at gmail.com
> > wrote:
>
>> Xsigo's SM is not "straight" OpenSM. They have some proprietary
>> enhancements and it may be based on old vintage of OpenSM. You will likely
>> need to work with them/Oracle now on issues.
>>
>> Lack of a partitions file does mean default partition and default rate
>> (10 Gbps) so from what I saw all ports had sufficient rate to join MC group.
>>
>> There are certain topology requirements for running various routing
>> algorithms. Did you try updn or dnup ?
>>
>> The key is determining whether the IPoIB broadcast group is setup
>> correctly. What MLID is the group built on (usually 0xc000) ? What are the
>> group parameters (rate, MTU) ? Are all members that are running IPoIB
>> joined ? Is the group routed to all such members ? There are
>> infiniband-diags for all of this.
>>
>>
>> On Mon, Oct 28, 2013 at 12:19 PM, Robert LeBlanc <robert_leblanc at byu.edu>wrote:
>>
>>> OpenSM (the SM runs on Xsigo so they manage it) is using minhop. I've
>>> loaded the ibnetdiscover output into ibsim and run all the different
>>> routing algorithms against it with and without scatter ports. Minhop had
>>> 50% of our hosts running all paths through a single IS5030 switch (at least
>>> the LIDs we need which represent Ethernet and Fibre Channel cards the hosts
>>> should communicate with). Ftree, dor, and dfsssp failed back to minhop, the
>>> others routed more paths through the same IS5030 in some cases increasing
>>> our host count with single point of failure to 75%.
>>>
>>> As far as I can tell there is no partitions.conf file so I assume we are
>>> using the default partition. There is an opensm.opts file, but it only
>>> specifies logging information.
>>> # SA database file name
>>> sa_db_file /var/log/opensm-sa.dump
>>>
>>> # If TRUE causes OpenSM to dump SA database at the end of
>>> # every light sweep, regardless of the verbosity level
>>> sa_db_dump TRUE
>>>
>>> # The directory to hold the file OpenSM dumps
>>> dump_files_dir /var/log/
>>>
>>> The SM node is:
>>> xsigoa:/opt/xsigo/xsigos/current/ofed/etc# ibaddr
>>> GID fe80::13:9702:100:979 LID start 0x1 end 0x1
>>>
>>> We do have Switch-X in two of the Dell m1000e chassis but the cards,
>>> ports 17-32, are FDR10 (the switch may be straight FDR, but I'm not 100%
>>> sure). The IS5030 are QDR which the Switch-X are connected to, the switches
>>> in the Xsigo directors are QDR, but the Ethernet and Fibre Channel cards
>>> are DDR. The DDR cards will not be running IPoIB (at least to my knowledge
>>> they don't have the ability), only the hosts should be leveraging IPoIB. I
>>> hope that clears up some of your questions. If you have more, I will try to
>>> answer them.
>>>
>>>
>>>
>>>
>>> Robert LeBlanc
>>> OIT Infrastructure & Virtualization Engineer
>>> Brigham Young University
>>>
>>>
>>> On Mon, Oct 28, 2013 at 9:57 AM, Hal Rosenstock <
>>> hal.rosenstock at gmail.com> wrote:
>>>
>>>> What routing algorithm is configured in OpenSM ? What does your
>>>> partitions.conf file look like ? Which node is your OpenSM ?
>>>>
>>>> Also, I only see QDR and DDR links although you have Switch-X so I
>>>> assume all FDR ports are connected to slower (QDR) devices. I don't see any
>>>> FDR-10 ports but maybe they're also connected to QDR ports so show up as
>>>> QDR in the topology.
>>>>
>>>> There are DDR CAs in Xsigo box but not sure whether or not they run
>>>> IPoIB.
>>>>
>>>> -- Hal
>>>>
>>>>
>>>> On Sun, Oct 27, 2013 at 9:46 PM, Robert LeBlanc <robert_leblanc at byu.edu
>>>> > wrote:
>>>>
>>>>> Since you guys are amazingly helpful, I thought I would pick your
>>>>> brains in a new problem.
>>>>>
>>>>> We have two Xsigo directors cross connected to four Mellanox IS5030
>>>>> switches. Connected to those we have four Dell m1000e chassis each with two
>>>>> IB switches (two chassis have QDR and two have FDR10). We have 9 dual-port
>>>>> rack servers connected to the IS5030 switches. For testing purposes we have
>>>>> an additional Dell m1000e QDR chassis connected to one Xsigo director and
>>>>> two dual-port FDR10 rack servers connected to the other Xsigo director.
>>>>>
>>>>> I can get IPoIB to work between the two test rack servers connected to
>>>>> the one Xsigo director. But I can not get IPoIB to work between any blades
>>>>> either right next to each other or to the working rack servers. I'm using
>>>>> the same exact live CentOS ISO on all four servers. I've checked opensm and
>>>>> the blades have joined the multicast group 0xc000 properly. tcpdump
>>>>> basically says that traffic is not leaving the blades. tcpdump also shows
>>>>> no traffic entering the blades from the rack servers. An ibtracert using
>>>>> 0xc000 mlid shows that routing exists between hosts.
>>>>>
>>>>> I've read about MulticastFDBTop=0xBFFF but I don't know how to set it
>>>>> and I doubt it would have been set by default.
>>>>>
>>>>> Anyone have some ideas on troubleshooting steps to try? I think Google
>>>>> is tired of me asking questions about it.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Robert LeBlanc
>>>>> OIT Infrastructure & Virtualization Engineer
>>>>> Brigham Young University
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users at lists.openfabrics.org
>>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/users/attachments/20131028/20054d1a/attachment.html>