[Users] IB topology config and polling state

Hal Rosenstock hal.rosenstock at gmail.com
Wed Oct 7 12:00:18 PDT 2015 

One more thing hopefully before playing with the low level phy settings:

Are you using known good cables ? Do you have FDR cables on the FDR <-> FDR
links ? Cable lengths can matter as well.

On Wed, Oct 7, 2015 at 12:57 PM, Hal Rosenstock <hal.rosenstock at gmail.com>
wrote:

> Were the ports mapped to the phy profile shutdown when you changed this ?
>
> LLR is a proprietary Mellanox mechanism.
>
> You might want 2 different profiles: one for the interfaces connected to
> other gateway interfaces (which are FDR (and FDR-10) capable and the other
> for the interfaces connecting to QDR (the older equipment in your network).
> By configuring the Switch-X interfaces to the appropriate possible speeds
> and disabling the proprietary mechanisms there, the link should not only
> come up but also this will occur faster than if FDR/FDR10 are enabled.
>
> I suspect that due to the Switch-X configuration that the links to
> the switch(es) in the HP enclosures do not negotiate properly (as shown by
> down rather than LinkUp).
>
> Once you get all your links to INIT, negotiation has occurred and then
> it's time for SM to bring links to active.
>
> Since you have down links, the SM can't do anything about those.
>
>
> On Wed, Oct 7, 2015 at 12:44 PM, German Anders <ganders at despegar.com>
> wrote:
>
>> Anyone had any experience with HP BLc 4X QDR IB Switch?? I know that this
>> kind of SW does not come with an embedded sm, but I don't know how to
>> access any mgmt at all on this particularly switch, I mean for example to
>> setup speed or anything like that, is possible to access through the
>> chassis?
>>
>>
>> *German* <ganders at despegar.com>
>>
>> 2015-10-07 13:19 GMT-03:00 German Anders <ganders at despegar.com>:
>>
>>> I think so, but when trying to configured the phy-profile on the
>>> interface in order to negotiate on QDR it failed to map the profile:
>>>
>>> GWIB01 [proxy-ha-group: master] (config) # show phy-profile
>>> high-speed-ber
>>>
>>>   Profile: high-speed-ber
>>>   --------
>>>   llr support ib-speed
>>>   SDR: disable
>>>   DDR: disable
>>>   QDR: disable
>>>   FDR10: enable-request
>>>   FDR: enable-request
>>>
>>> GWIB01 [proxy-ha-group: master] (config) # show phy-profile hp-encl-isl
>>>
>>>   Profile: hp-encl-isl
>>>   --------
>>>   llr support ib-speed
>>>   SDR: disable
>>>   DDR: disable
>>>   QDR: enable
>>>   FDR10: enable-request
>>>   FDR: enable-request
>>>
>>> GWIB01 [proxy-ha-group: master] (config) #
>>> GWIB01 [proxy-ha-group: master] (config) # interface ib 1/9 phy-profile
>>> map hp-encl-isl
>>> *% Cannot map profile hp-encl-isl to port:  1/9*
>>>
>>>
>>> *German* <ganders at despegar.com>
>>>
>>> 2015-10-07 13:17 GMT-03:00 Weiny, Ira <ira.weiny at intel.com>:
>>>
>>>> The driver ‘qib’ is loading fine.  As can be seen by the ibstat
>>>> output.  The ib_ipath is an older card.
>>>>
>>>>
>>>>
>>>> The problem is the link is not coming up to init.  Like Hal said the
>>>> link should transition to “link up” without the SMs involvement.
>>>>
>>>>
>>>>
>>>> I think you are on to something with the fact that it seems like your
>>>> switch ports are not configured to do QDR.
>>>>
>>>>
>>>>
>>>> Ira
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *From:* German Anders [mailto:ganders at despegar.com]
>>>> *Sent:* Wednesday, October 07, 2015 9:05 AM
>>>> *To:* Weiny, Ira
>>>> *Cc:* Hal Rosenstock; users at lists.openfabrics.org
>>>>
>>>> *Subject:* Re: [Users] IB topology config and polling state
>>>>
>>>>
>>>>
>>>> Yes I've that file:
>>>>
>>>> /usr/sbin/truescale-serdes.cmds
>>>>
>>>> Also I've done the install of libipathverbs:
>>>>
>>>> # apt-get install libipathverbs-dev
>>>>
>>>> But I try to load the ib_ipath module but I'm getting the following
>>>> error msg:
>>>>
>>>> # modprobe ib_ipath
>>>> modprobe: ERROR: could not insert 'ib_ipath': Device or resource busy
>>>>
>>>>
>>>>
>>>> *German*
>>>>
>>>>
>>>>
>>>> 2015-10-07 12:54 GMT-03:00 Weiny, Ira <ira.weiny at intel.com>:
>>>>
>>>> There are a few issues for routing in that diagram but the links should
>>>> come up.
>>>>
>>>>
>>>>
>>>> I assume there is some backplane between the blade servers and the
>>>> switch in that chassis?
>>>>
>>>>
>>>>
>>>> Have you gotten libipathverbs installed?
>>>>
>>>>
>>>>
>>>> In ipathverbs there is a serdes tuning script.
>>>>
>>>>
>>>>
>>>> https://github.com/01org/libipathverbs/blob/master/truescale-serdes.cmds
>>>>
>>>>
>>>>
>>>> Does your libipathverbs include that file?  If not try the latest from
>>>> github.
>>>>
>>>>
>>>>
>>>> Ira
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *From:* users-bounces at lists.openfabrics.org [mailto:
>>>> users-bounces at lists.openfabrics.org] *On Behalf Of *German Anders
>>>> *Sent:* Wednesday, October 07, 2015 8:41 AM
>>>> *To:* Hal Rosenstock
>>>> *Cc:* users at lists.openfabrics.org
>>>> *Subject:* Re: [Users] IB topology config and polling state
>>>>
>>>>
>>>>
>>>> Hi Hal,
>>>>
>>>> Thanks for the reply, I've attach a pdf with the diagram topology, I
>>>> don't know if this is the best way to go or if there's another way to
>>>> connect and setup the IB network, tips and suggestions will be very
>>>> appreciated, also the mezzanine cards are already installed on the blade
>>>> hosts:
>>>>
>>>> # lspci
>>>> (...)
>>>> 41:00.0 InfiniBand: QLogic Corp. IBA7322 QDR InfiniBand HCA (rev 02)
>>>>
>>>>
>>>>
>>>> Thanks in advance,
>>>>
>>>> Cheers,
>>>>
>>>>
>>>> *German*
>>>>
>>>>
>>>>
>>>> 2015-10-07 11:47 GMT-03:00 Hal Rosenstock <hal.rosenstock at gmail.com>:
>>>>
>>>> Hi again German,
>>>>
>>>>
>>>>
>>>> Looks like you made some progress from yesterday as the qib ports are
>>>> now Polling rather than Disabled.
>>>>
>>>>
>>>>
>>>> But since they are Down, do you have them cabled to a switch ? That
>>>> should bring the links up and the port state will be Init. That is the
>>>> "starting" point.
>>>>
>>>>
>>>>
>>>> You will also then need to be running SM to bring the ports up to
>>>> Active.
>>>>
>>>>
>>>>
>>>> -- Hal
>>>>
>>>>
>>>>
>>>> On Wed, Oct 7, 2015 at 10:37 AM, German Anders <ganders at despegar.com>
>>>> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I don't know if this is the mailist list for this kind of topic but I'm
>>>> really new to IB and I've just install two SX6036G gateways connected to
>>>> each other through two ISL ports, then I've configured a proxy-arp between
>>>> both nodes (sm is disable on both gw's):
>>>>
>>>> GWIB01 [proxy-ha-group: master] (config) # show proxy-arp ha
>>>>
>>>> Load balancing algorithm: ib-base-ip
>>>> Number of Proxy-Arp interfaces: 1
>>>>
>>>> Proxy-ARP VIP
>>>> =============
>>>> Pra-group name: proxy-ha-group
>>>> HA VIP address: 10.xx.xx.xx/xx
>>>>
>>>> Active nodes:
>>>> ID                   State                IP
>>>> --------------------------------------------------------------
>>>> GWIB01               master               10.xx.xx.xx1
>>>> GWIB02               standby              10.xx.xx.xx2
>>>>
>>>> Then I setup two SX6018F switches (*SWIB01* and *SWIB02*), one
>>>> connected to GWIB01 and the other connected to GWIB02. The SM is configured
>>>> locally on both SWIB01 & SWIB02 switches. So far so good, after this config
>>>> I setup a commodity server with a MLNX IB ADPT FDR to the SWIB01 & SWIB02
>>>> switches, config the drivers, etc and then get it up & running fine.
>>>>
>>>> Finally I've setup a HP Enclosure with an internal IB SW (then connect
>>>> port 1 of the internal SW to GWIB01 - link is up but LLR status is
>>>> inactive), install one of the blades and I see the following:
>>>>
>>>> # ibstat
>>>> CA 'qib0'
>>>>     CA type: InfiniPath_QMH7342
>>>>     Number of ports: 2
>>>>     Firmware version:
>>>>     Hardware version: 2
>>>>     Node GUID: 0x0011750000791fec
>>>>     System image GUID: 0x0011750000791fec
>>>>     Port 1:
>>>>         State: Down
>>>>         Physical state: Polling
>>>>         Rate: 40
>>>>         Base lid: 4660
>>>>         LMC: 0
>>>>         SM lid: 4660
>>>>         Capability mask: 0x0761086a
>>>>         Port GUID: 0x0011750000791fec
>>>>         Link layer: InfiniBand
>>>>     Port 2:
>>>>         State: Down
>>>>         Physical state: Polling
>>>>         Rate: 40
>>>>         Base lid: 4660
>>>>         LMC: 0
>>>>         SM lid: 4660
>>>>         Capability mask: 0x0761086a
>>>>         Port GUID: 0x0011750000791fed
>>>>         Link layer: InfiniBand
>>>>
>>>> So I was wondering if maybe the SM is not being recognized on the Blade
>>>> system and that's why is not passing the Polling state, is that possible?
>>>> Or maybe is not possible to connect an ISL between the GW and the HP
>>>> internal SW so that the sm is available or maybe the inactive LLR is
>>>> causing this thing, any ideas? I thought about connecting the ISL of
>>>> the HP IB SW to the SWIB01 or SWIB02 instead of the GW's but I don't have
>>>> any available ports.
>>>>
>>>> Thanks in advance,
>>>>
>>>> Cheers,
>>>>
>>>>
>>>> *German*
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at lists.openfabrics.org
>>>> http://lists.openfabrics.org/mailman/listinfo/users
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/users/attachments/20151007/93218208/attachment.html>

More information about the Users mailing list