[Users] IB topology config and polling state
German Anders
ganders at despegar.com
Tue Oct 13 04:44:54 PDT 2015
Hi Ira,
I've some HP documentation but it quite short, also it doesn't describe any
'config' or 'impl' steps in order to get the internal switch up and
running. The version of SW that came with enclosure does not had any
management module at all, so it depends on external management. During
weekend I've found a way to upgrade the firmware of the HP switch with the
following command (mlxburn -d lid-0x001D -fw fw-IS4.mlx), and the I've run
(flint -d /dev/mst/SW_MT48438_0x2c902004b0918_lid-0x001D dc
/home/ceph/HPIBSW.INI) and found the following inside that file:
[PS_INFO]
Name = 489184-B21
Description = HP BLc 4X QDR IB Switch
[ADAPTER]
PSID = HP_0100000009
(...)
[IB_TO_HW_MAP]
PORT1=14
PORT2=15
PORT3=16
PORT4=17
PORT5=18
PORT6=12
PORT7=11
PORT8=10
PORT9=9
PORT10=8
PORT11=7
PORT12=6
PORT13=5
PORT14=4
PORT15=3
PORT16=2
PORT17=20
PORT18=22
PORT19=24
PORT20=26
PORT21=28
PORT22=30
PORT23=35
PORT24=33
PORT25=21
PORT26=23
PORT27=25
PORT28=27
PORT29=29
PORT30=36
PORT31=34
PORT32=32
PORT33=1
PORT34=13
PORT35=19
PORT36=31
[unused_ports]
hw_port1_not_in_use=1
hw_port13_not_in_use=1
hw_port19_not_in_use=1
hw_port31_not_in_use=1
(...)
I don't know if maybe there's some issue with the port mapping, anyone had
used this kind of switch?
The summary of the problem is correct, the connectivity between the IB
network (MLNX switches/gw) and the HP IB switch is working since I was able
to upgrade the firmare of the switch and get information about it. But, the
connection between the mezzanine cards of the blades and the internal IB sw
enclosure is not working at all. Note, that if I go to the OA
administration of the enclosure I can see the 'green' ports mapping of each
of the blades and the interconnection switch, so I'm guessing that it
should be working.
Regarding the questions:
1) What type of switch is in the HP chassis?
*QLogic HP BLc 4X QDR IB Switch*
*PSID = HP_0100000009*
*Image type: FS2*
*FW ver: 7.4.3000*
*Device ID: 48438*
*GUI: 0002c902004b0918*
2) Do you have console access or http access to that switch?
*No, since it didn't had any manage module mezzanine card inside the
switch, it only come with a i2c port. But, i can have access through the
mlxburn and flint tools from one host that's connected to the ib network
(outside the enclosure).*
3) Does that switch have an SM in it?
*No*
4) What version of the kernel are you running with the qib cards?
a. I assume you are using the qib driver in that kernel.
*Ubuntu 14.04.3 LTS - kernel 3.18.20-031820-generic*
At some point Hal spoke of “LLR being a Mellanox thing” Was that to solve
the problem of connecting the “HP switch” to the Mellanox switch?
*No, since LLR is only supported between mlnx devices, the ISL are up and
working, since it's possible for me to query the switch*
I would like it if you could verify that the
/usr/sbin/truescale-serdes.cmds
Is being run?
*When trying to run the command:*
*# /usr/sbin/truescale-serdes.cmds/usr/sbin/truescale-serdes.cmds: 100:
/usr/sbin/truescale-serdes.cmds: Syntax error: "(" unexpected (expecting
"}")*
Also what version of libipathverbs do you have?
*# rpm -qa | grep libipathverbslibipathverbs-1.3-1.x86_64*
Thanks in advance,
Cheers,
*German*
2015-10-13 2:14 GMT-03:00 Weiny, Ira <ira.weiny at intel.com>:
> German,
>
>
>
> Do you have any documentation on the HP blade system? And the switch
> which is in that system?
>
>
>
> I have to admit I have not followed everything in this thread regarding
> your configuration but it seems like you have some mellanox switches
> connected into an HP chassis which has both a switch and blades with qib
> (Truescale) cards.
>
>
>
> The connection from the mellanox switch to the “HP chassis switch” is
> linkup (active) but the connections to the individual qib HCAs are not even
> linkup.
>
>
>
> Is that a correct summary of the problem?
>
>
>
> If so here are some questions:
>
>
>
> 1) What type of switch is in the HP chassis?
>
> 2) Do you have console access or http access to that switch?
>
> 3) Does that switch have an SM in it?
>
> 4) What version of the kernel are you running with the qib cards?
>
> a. I assume you are using the qib driver in that kernel.
>
>
>
> At some point Hal spoke of “LLR being a Mellanox thing” Was that to solve
> the problem of connecting the “HP switch” to the Mellanox switch?
>
>
>
> I would like it if you could verify that the
>
>
>
> /usr/sbin/truescale-serdes.cmds
>
>
>
> Is being run?
>
>
>
> Also what version of libipathverbs do you have?
>
>
>
> Ira
>
>
>
> *From:* users-bounces at lists.openfabrics.org [mailto:
> users-bounces at lists.openfabrics.org] *On Behalf Of *Weiny, Ira
> *Sent:* Wednesday, October 07, 2015 1:31 PM
> *To:* Hal Rosenstock; German Anders
>
> *Cc:* users at lists.openfabrics.org
> *Subject:* Re: [Users] IB topology config and polling state
>
>
>
> Agree with Hal here.
>
>
>
> I’m not familiar with those blades/switches. I’ll ask around.
>
>
>
> Ira
>
>
>
> *From:* Hal Rosenstock [mailto:hal.rosenstock at gmail.com
> <hal.rosenstock at gmail.com>]
> *Sent:* Wednesday, October 07, 2015 1:26 PM
> *To:* German Anders
> *Cc:* Weiny, Ira; users at lists.openfabrics.org
> *Subject:* Re: [Users] IB topology config and polling state
>
>
>
> That's the gateway to the switch in the enclosure. It's the internal
> connectivity in the blade enclosure that's (physically) broken.
>
>
>
> On Wed, Oct 7, 2015 at 4:24 PM, German Anders <ganders at despegar.com>
> wrote:
>
> cabled
>
> the blade it's:
>
> vendid=0x2c9
> devid=0xbd36
> sysimgguid=0x2c902004b0918
> switchguid=0x2c902004b0918(2c902004b0918)
> Switch 32 "S-0002c902004b0918" # "Infiniscale-IV Mellanox
> Technologies" base port 0 *lid 29* lmc 0
> [1] "S-e41d2d030031e9c1"[9] # "MF0;GWIB01:SX6036G/U1" lid 24
> 4xQDR
>
>
> *German*
>
>
>
> 2015-10-07 17:21 GMT-03:00 Hal Rosenstock <hal.rosenstock at gmail.com>:
>
> What are those HCAs cabled to or is it internal to the blade enclosure ?
>
>
>
> On Wed, Oct 7, 2015 at 3:24 PM, German Anders <ganders at despegar.com>
> wrote:
>
> Yeah, is there any command that I can run in order to change the port
> state on the remote switch? I mean everything looks good but in the hp
> blades still getting:
>
>
>
> # ibstat
> CA 'qib0'
> CA type: InfiniPath_QMH7342
> Number of ports: 2
> Firmware version:
> Hardware version: 2
> Node GUID: 0x0011750000791fec
> System image GUID: 0x0011750000791fec
> Port 1:
> State: *Down*
> Physical state: *Polling*
> Rate: 40
> Base lid: 4660
> LMC: 0
> SM lid: 4660
> Capability mask: 0x0761086a
> Port GUID: 0x0011750000791fec
> Link layer: InfiniBand
> Port 2:
> State: *Down*
> Physical state: *Polling*
> Rate: 40
> Base lid: 4660
> LMC: 0
> SM lid: 4660
> Capability mask: 0x0761086a
> Port GUID: 0x0011750000791fed
> Link layer: InfiniBand
>
> Also on working hosts I only see devices from the local network, but
> didn't see any of the blades hca connections.
>
>
>
>
> *German*
>
>
>
> 2015-10-07 16:21 GMT-03:00 Hal Rosenstock <hal.rosenstock at gmail.com>:
>
> The screen shot looks good :-) SM brought the link up to active.
>
>
>
> Note that the ibportstate command you gave was for switch port 0 of the
> Mellanox IS-4 switch in the QLogic HP BLc 4X QDR IB Switch.
>
>
>
> On Wed, Oct 7, 2015 at 3:06 PM, German Anders <ganders at despegar.com>
> wrote:
>
> Yes, find attached an screenshot of the port information (# 9) the one
> that makes the ISL to the QLogic HP BLc 4X QDR IB Switch, also from one of
> the hosts that are connected to one of the SX6018F I can see the 'remote'
> HP IB SW:
>
> # *ibnodes*
>
> (...)
> Switch : 0x0002c902004b0918 ports 32 "Infiniscale-IV Mellanox
> Technologies" base port 0 *lid 29* lmc 0
> Switch : 0xe41d2d030031e9c1 ports 37 "MF0;GWIB01:SX6036G/U1" enhanced
> port 0 lid 24 lmc 0
> (...)
>
> # *ibportstate -L 29 query*
> Switch PortInfo:
> # Port info: Lid 29 port 0
> LinkState:.......................Active
> PhysLinkState:...................LinkUp
> Lid:.............................29
> SMLid:...........................2
> LMC:.............................0
> LinkWidthSupported:..............1X or 4X
> LinkWidthEnabled:................1X or 4X
> LinkWidthActive:.................4X
> LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
> LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
> LinkSpeedActive:.................10.0 Gbps
> Mkey:............................<not displayed>
> MkeyLeasePeriod:.................0
> ProtectBits:.....................0
>
>
>
>
> *German*
>
>
>
> 2015-10-07 16:00 GMT-03:00 Hal Rosenstock <hal.rosenstock at gmail.com>:
>
> One more thing hopefully before playing with the low level phy settings:
>
>
>
> Are you using known good cables ? Do you have FDR cables on the FDR <->
> FDR links ? Cable lengths can matter as well.
>
>
>
> On Wed, Oct 7, 2015 at 12:57 PM, Hal Rosenstock <hal.rosenstock at gmail.com>
> wrote:
>
> Were the ports mapped to the phy profile shutdown when you changed this ?
>
>
>
> LLR is a proprietary Mellanox mechanism.
>
>
>
> You might want 2 different profiles: one for the interfaces connected to
> other gateway interfaces (which are FDR (and FDR-10) capable and the other
> for the interfaces connecting to QDR (the older equipment in your network).
> By configuring the Switch-X interfaces to the appropriate possible speeds
> and disabling the proprietary mechanisms there, the link should not only
> come up but also this will occur faster than if FDR/FDR10 are enabled.
>
>
>
> I suspect that due to the Switch-X configuration that the links to
> the switch(es) in the HP enclosures do not negotiate properly (as shown by
> down rather than LinkUp).
>
>
>
> Once you get all your links to INIT, negotiation has occurred and then
> it's time for SM to bring links to active.
>
>
>
> Since you have down links, the SM can't do anything about those.
>
>
>
>
>
> On Wed, Oct 7, 2015 at 12:44 PM, German Anders <ganders at despegar.com>
> wrote:
>
> Anyone had any experience with HP BLc 4X QDR IB Switch?? I know that this
> kind of SW does not come with an embedded sm, but I don't know how to
> access any mgmt at all on this particularly switch, I mean for example to
> setup speed or anything like that, is possible to access through the
> chassis?
>
>
> *German*
>
>
>
> 2015-10-07 13:19 GMT-03:00 German Anders <ganders at despegar.com>:
>
> I think so, but when trying to configured the phy-profile on the interface
> in order to negotiate on QDR it failed to map the profile:
>
> GWIB01 [proxy-ha-group: master] (config) # show phy-profile high-speed-ber
>
> Profile: high-speed-ber
> --------
> llr support ib-speed
> SDR: disable
> DDR: disable
> QDR: disable
> FDR10: enable-request
> FDR: enable-request
>
> GWIB01 [proxy-ha-group: master] (config) # show phy-profile hp-encl-isl
>
> Profile: hp-encl-isl
> --------
> llr support ib-speed
> SDR: disable
> DDR: disable
> QDR: enable
> FDR10: enable-request
> FDR: enable-request
>
> GWIB01 [proxy-ha-group: master] (config) #
> GWIB01 [proxy-ha-group: master] (config) # interface ib 1/9 phy-profile
> map hp-encl-isl
> *% Cannot map profile hp-encl-isl to port: 1/9*
>
>
> *German*
>
>
>
> 2015-10-07 13:17 GMT-03:00 Weiny, Ira <ira.weiny at intel.com>:
>
> The driver ‘qib’ is loading fine. As can be seen by the ibstat output.
> The ib_ipath is an older card.
>
>
>
> The problem is the link is not coming up to init. Like Hal said the link
> should transition to “link up” without the SMs involvement.
>
>
>
> I think you are on to something with the fact that it seems like your
> switch ports are not configured to do QDR.
>
>
>
> Ira
>
>
>
>
>
> *From:* German Anders [mailto:ganders at despegar.com]
> *Sent:* Wednesday, October 07, 2015 9:05 AM
> *To:* Weiny, Ira
> *Cc:* Hal Rosenstock; users at lists.openfabrics.org
>
>
> *Subject:* Re: [Users] IB topology config and polling state
>
>
>
> Yes I've that file:
>
> /usr/sbin/truescale-serdes.cmds
>
> Also I've done the install of libipathverbs:
>
> # apt-get install libipathverbs-dev
>
> But I try to load the ib_ipath module but I'm getting the following error
> msg:
>
> # modprobe ib_ipath
> modprobe: ERROR: could not insert 'ib_ipath': Device or resource busy
>
>
> *German*
>
>
>
> 2015-10-07 12:54 GMT-03:00 Weiny, Ira <ira.weiny at intel.com>:
>
> There are a few issues for routing in that diagram but the links should
> come up.
>
>
>
> I assume there is some backplane between the blade servers and the switch
> in that chassis?
>
>
>
> Have you gotten libipathverbs installed?
>
>
>
> In ipathverbs there is a serdes tuning script.
>
>
>
> https://github.com/01org/libipathverbs/blob/master/truescale-serdes.cmds
>
>
>
> Does your libipathverbs include that file? If not try the latest from
> github.
>
>
>
> Ira
>
>
>
>
>
> *From:* users-bounces at lists.openfabrics.org [mailto:
> users-bounces at lists.openfabrics.org] *On Behalf Of *German Anders
> *Sent:* Wednesday, October 07, 2015 8:41 AM
> *To:* Hal Rosenstock
> *Cc:* users at lists.openfabrics.org
> *Subject:* Re: [Users] IB topology config and polling state
>
>
>
> Hi Hal,
>
> Thanks for the reply, I've attach a pdf with the diagram topology, I don't
> know if this is the best way to go or if there's another way to connect and
> setup the IB network, tips and suggestions will be very appreciated, also
> the mezzanine cards are already installed on the blade hosts:
>
> # lspci
> (...)
> 41:00.0 InfiniBand: QLogic Corp. IBA7322 QDR InfiniBand HCA (rev 02)
>
>
>
> Thanks in advance,
>
> Cheers,
>
>
> *German*
>
>
>
> 2015-10-07 11:47 GMT-03:00 Hal Rosenstock <hal.rosenstock at gmail.com>:
>
> Hi again German,
>
>
>
> Looks like you made some progress from yesterday as the qib ports are now
> Polling rather than Disabled.
>
>
>
> But since they are Down, do you have them cabled to a switch ? That should
> bring the links up and the port state will be Init. That is the "starting"
> point.
>
>
>
> You will also then need to be running SM to bring the ports up to Active.
>
>
>
> -- Hal
>
>
>
> On Wed, Oct 7, 2015 at 10:37 AM, German Anders <ganders at despegar.com>
> wrote:
>
> Hi all,
>
> I don't know if this is the mailist list for this kind of topic but I'm
> really new to IB and I've just install two SX6036G gateways connected to
> each other through two ISL ports, then I've configured a proxy-arp between
> both nodes (sm is disable on both gw's):
>
> GWIB01 [proxy-ha-group: master] (config) # show proxy-arp ha
>
> Load balancing algorithm: ib-base-ip
> Number of Proxy-Arp interfaces: 1
>
> Proxy-ARP VIP
> =============
> Pra-group name: proxy-ha-group
> HA VIP address: 10.xx.xx.xx/xx
>
> Active nodes:
> ID State IP
> --------------------------------------------------------------
> GWIB01 master 10.xx.xx.xx1
> GWIB02 standby 10.xx.xx.xx2
>
> Then I setup two SX6018F switches (*SWIB01* and *SWIB02*), one connected
> to GWIB01 and the other connected to GWIB02. The SM is configured locally
> on both SWIB01 & SWIB02 switches. So far so good, after this config I setup
> a commodity server with a MLNX IB ADPT FDR to the SWIB01 & SWIB02 switches,
> config the drivers, etc and then get it up & running fine.
>
> Finally I've setup a HP Enclosure with an internal IB SW (then connect
> port 1 of the internal SW to GWIB01 - link is up but LLR status is
> inactive), install one of the blades and I see the following:
>
> # ibstat
> CA 'qib0'
> CA type: InfiniPath_QMH7342
> Number of ports: 2
> Firmware version:
> Hardware version: 2
> Node GUID: 0x0011750000791fec
> System image GUID: 0x0011750000791fec
> Port 1:
> State: Down
> Physical state: Polling
> Rate: 40
> Base lid: 4660
> LMC: 0
> SM lid: 4660
> Capability mask: 0x0761086a
> Port GUID: 0x0011750000791fec
> Link layer: InfiniBand
> Port 2:
> State: Down
> Physical state: Polling
> Rate: 40
> Base lid: 4660
> LMC: 0
> SM lid: 4660
> Capability mask: 0x0761086a
> Port GUID: 0x0011750000791fed
> Link layer: InfiniBand
>
> So I was wondering if maybe the SM is not being recognized on the Blade
> system and that's why is not passing the Polling state, is that possible?
> Or maybe is not possible to connect an ISL between the GW and the HP
> internal SW so that the sm is available or maybe the inactive LLR is
> causing this thing, any ideas? I thought about connecting the ISL of the
> HP IB SW to the SWIB01 or SWIB02 instead of the GW's but I don't have any
> available ports.
>
> Thanks in advance,
>
> Cheers,
>
>
> *German*
>
>
>
> _______________________________________________
> Users mailing list
> Users at lists.openfabrics.org
> http://lists.openfabrics.org/mailman/listinfo/users
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/users/attachments/20151013/6afa5102/attachment.html>
More information about the Users
mailing list