[Users] IB topology config and polling state
German Anders
ganders at despegar.com
Tue Oct 13 06:51:37 PDT 2015
yes, but the switch does not come with an external interface... just the
i2c port :S
*German* <ganders at despegar.com>
2015-10-13 10:49 GMT-03:00 Hal Rosenstock <hal.rosenstock at gmail.com>:
> I don't really know but I was wondering about whether your hardware
> supports this.
>
> HPOA_2 shows internal interface to OA is absent as is external Ethernet
> interface.
>
> On Tue, Oct 13, 2015 at 9:31 AM, German Anders <ganders at despegar.com>
> wrote:
>
>> when trying to assign the ip addr to the interconnect bay it seems that
>> it does not 'apply' the change, and the ip addr is not displayed as
>> 'configured' so there's no way to enter from outside. ideas?
>>
>> *German*
>>
>> 2015-10-13 10:15 GMT-03:00 Hal Rosenstock <hal.rosenstock at gmail.com>:
>>
>>> What about the 3 critical errors ? What are they ?
>>>
>>> On Tue, Oct 13, 2015 at 9:13 AM, German Anders <ganders at despegar.com>
>>> wrote:
>>>
>>>> I've try that and in fact I try to put the IP ADDR like this:
>>>>
>>>> SET EBIPA INTERCONNECT XX.XX.XX.XX XXX.XXX.XXX.XXX 3
>>>> SET EBIPA INTERCONNECT GATEWAY XX.XX.XX.XX 3
>>>> SET EBIPA INTERCONNECT DOMAIN "xxxxxx.net" 3
>>>> ADD EBIPA INTERCONNECT DNS 10.xx.xx.xx 3
>>>> ADD EBIPA INTERCONNECT DNS 10.xx.xx.xx 3
>>>> SET EBIPA INTERCONNECT NTP PRIMARY NONE 3
>>>> SET EBIPA INTERCONNECT NTP SECONDARY NONE 3
>>>> ENABLE EBIPA INTERCONNECT 3
>>>>
>>>> SAVE EBIPA
>>>>
>>>> But i'm not getting any ip response, also I've try many diff ip addr with no luck...if i put that ip to one of the blades it works fine, but not to the interconnect bay :( any other idea?
>>>>
>>>> Cheers,
>>>>
>>>>
>>>> *German* <ganders at despegar.com>
>>>>
>>>> 2015-10-13 10:01 GMT-03:00 Hal Rosenstock <hal.rosenstock at gmail.com>:
>>>>
>>>>> Looks like there are 3 critical errors in system status. Did you look
>>>>> at these ?
>>>>>
>>>>> I don't know if you've seen this but there is some info on configuring
>>>>> the management IPs in
>>>>> http://h10032.www1.hp.com/ctg/Manual/c00814176.pdf
>>>>>
>>>>> Have you looked at/tried the command line interface ?
>>>>>
>>>>> On Tue, Oct 13, 2015 at 8:28 AM, German Anders <ganders at despegar.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Hal,
>>>>>>
>>>>>> It does not allow me to setup an IP ADDR to the Internal SW so I
>>>>>> can't access from outside, except from the tools that I mentioned before,
>>>>>> also it doesn't allow me to access through serial connection from inside
>>>>>> the enclosure. I've attach some screen-shots about the connectivity.
>>>>>>
>>>>>>
>>>>>>
>>>>>> *German*
>>>>>>
>>>>>> 2015-10-13 9:13 GMT-03:00 Hal Rosenstock <hal.rosenstock at gmail.com>:
>>>>>>
>>>>>>> Hi German,
>>>>>>>
>>>>>>> Are the cards in the correct bays and slots ?
>>>>>>>
>>>>>>> Do you have the HP Onboard Administrator tool ? What does it say
>>>>>>> about internal connectivity ?
>>>>>>>
>>>>>>> -- Hal
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Oct 13, 2015 at 7:44 AM, German Anders <ganders at despegar.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hi Ira,
>>>>>>>>
>>>>>>>> I've some HP documentation but it quite short, also it doesn't
>>>>>>>> describe any 'config' or 'impl' steps in order to get the internal switch
>>>>>>>> up and running. The version of SW that came with enclosure does not had any
>>>>>>>> management module at all, so it depends on external management. During
>>>>>>>> weekend I've found a way to upgrade the firmware of the HP switch with the
>>>>>>>> following command (mlxburn -d lid-0x001D -fw fw-IS4.mlx), and the I've run
>>>>>>>> (flint -d /dev/mst/SW_MT48438_0x2c902004b0918_lid-0x001D dc
>>>>>>>> /home/ceph/HPIBSW.INI) and found the following inside that file:
>>>>>>>>
>>>>>>>> [PS_INFO]
>>>>>>>> Name = 489184-B21
>>>>>>>> Description = HP BLc 4X QDR IB Switch
>>>>>>>>
>>>>>>>> [ADAPTER]
>>>>>>>> PSID = HP_0100000009
>>>>>>>>
>>>>>>>> (...)
>>>>>>>>
>>>>>>>> [IB_TO_HW_MAP]
>>>>>>>> PORT1=14
>>>>>>>> PORT2=15
>>>>>>>> PORT3=16
>>>>>>>> PORT4=17
>>>>>>>> PORT5=18
>>>>>>>> PORT6=12
>>>>>>>> PORT7=11
>>>>>>>> PORT8=10
>>>>>>>> PORT9=9
>>>>>>>> PORT10=8
>>>>>>>> PORT11=7
>>>>>>>> PORT12=6
>>>>>>>> PORT13=5
>>>>>>>> PORT14=4
>>>>>>>> PORT15=3
>>>>>>>> PORT16=2
>>>>>>>> PORT17=20
>>>>>>>> PORT18=22
>>>>>>>> PORT19=24
>>>>>>>>
>>>>>>>> PORT20=26
>>>>>>>> PORT21=28
>>>>>>>> PORT22=30
>>>>>>>> PORT23=35
>>>>>>>> PORT24=33
>>>>>>>> PORT25=21
>>>>>>>> PORT26=23
>>>>>>>> PORT27=25
>>>>>>>> PORT28=27
>>>>>>>> PORT29=29
>>>>>>>> PORT30=36
>>>>>>>> PORT31=34
>>>>>>>> PORT32=32
>>>>>>>> PORT33=1
>>>>>>>> PORT34=13
>>>>>>>> PORT35=19
>>>>>>>> PORT36=31
>>>>>>>>
>>>>>>>> [unused_ports]
>>>>>>>> hw_port1_not_in_use=1
>>>>>>>> hw_port13_not_in_use=1
>>>>>>>> hw_port19_not_in_use=1
>>>>>>>> hw_port31_not_in_use=1
>>>>>>>>
>>>>>>>> (...)
>>>>>>>>
>>>>>>>> I don't know if maybe there's some issue with the port mapping,
>>>>>>>> anyone had used this kind of switch?
>>>>>>>>
>>>>>>>> The summary of the problem is correct, the connectivity between the
>>>>>>>> IB network (MLNX switches/gw) and the HP IB switch is working since I was
>>>>>>>> able to upgrade the firmare of the switch and get information about it.
>>>>>>>> But, the connection between the mezzanine cards of the blades and the
>>>>>>>> internal IB sw enclosure is not working at all. Note, that if I go to the
>>>>>>>> OA administration of the enclosure I can see the 'green' ports mapping of
>>>>>>>> each of the blades and the interconnection switch, so I'm guessing that it
>>>>>>>> should be working.
>>>>>>>>
>>>>>>>> Regarding the questions:
>>>>>>>>
>>>>>>>> 1) What type of switch is in the HP chassis?
>>>>>>>>
>>>>>>>>
>>>>>>>> *QLogic HP BLc 4X QDR IB Switch*
>>>>>>>>
>>>>>>>> *PSID = HP_0100000009*
>>>>>>>>
>>>>>>>> *Image type: FS2*
>>>>>>>>
>>>>>>>> *FW ver: 7.4.3000*
>>>>>>>>
>>>>>>>> *Device ID: 48438*
>>>>>>>> *GUI: 0002c902004b0918*
>>>>>>>>
>>>>>>>> 2) Do you have console access or http access to that switch?
>>>>>>>>
>>>>>>>> *No, since it didn't had any manage module mezzanine card inside
>>>>>>>> the switch, it only come with a i2c port. But, i can have access through
>>>>>>>> the mlxburn and flint tools from one host that's connected to the ib
>>>>>>>> network (outside the enclosure).*
>>>>>>>>
>>>>>>>> 3) Does that switch have an SM in it?
>>>>>>>>
>>>>>>>> *No*
>>>>>>>>
>>>>>>>> 4) What version of the kernel are you running with the qib
>>>>>>>> cards?
>>>>>>>>
>>>>>>>> a. I assume you are using the qib driver in that kernel.
>>>>>>>>
>>>>>>>> *Ubuntu 14.04.3 LTS - kernel 3.18.20-031820-generic*
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> At some point Hal spoke of “LLR being a Mellanox thing” Was that
>>>>>>>> to solve the problem of connecting the “HP switch” to the Mellanox switch?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *No, since LLR is only supported between mlnx devices, the ISL are
>>>>>>>> up and working, since it's possible for me to query the switch*
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I would like it if you could verify that the
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> /usr/sbin/truescale-serdes.cmds
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Is being run?
>>>>>>>>
>>>>>>>>
>>>>>>>> *When trying to run the command:*
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *# /usr/sbin/truescale-serdes.cmds/usr/sbin/truescale-serdes.cmds:
>>>>>>>> 100: /usr/sbin/truescale-serdes.cmds: Syntax error: "(" unexpected
>>>>>>>> (expecting "}")*
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Also what version of libipathverbs do you have?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *# rpm -qa | grep libipathverbslibipathverbs-1.3-1.x86_64*
>>>>>>>> Thanks in advance,
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *German*
>>>>>>>> 2015-10-13 2:14 GMT-03:00 Weiny, Ira <ira.weiny at intel.com>:
>>>>>>>>
>>>>>>>>> German,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Do you have any documentation on the HP blade system? And the
>>>>>>>>> switch which is in that system?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have to admit I have not followed everything in this thread
>>>>>>>>> regarding your configuration but it seems like you have some mellanox
>>>>>>>>> switches connected into an HP chassis which has both a switch and blades
>>>>>>>>> with qib (Truescale) cards.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The connection from the mellanox switch to the “HP chassis switch”
>>>>>>>>> is linkup (active) but the connections to the individual qib HCAs are not
>>>>>>>>> even linkup.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Is that a correct summary of the problem?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> If so here are some questions:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 1) What type of switch is in the HP chassis?
>>>>>>>>>
>>>>>>>>> 2) Do you have console access or http access to that switch?
>>>>>>>>>
>>>>>>>>> 3) Does that switch have an SM in it?
>>>>>>>>>
>>>>>>>>> 4) What version of the kernel are you running with the qib
>>>>>>>>> cards?
>>>>>>>>>
>>>>>>>>> a. I assume you are using the qib driver in that kernel.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> At some point Hal spoke of “LLR being a Mellanox thing” Was that
>>>>>>>>> to solve the problem of connecting the “HP switch” to the Mellanox switch?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I would like it if you could verify that the
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> /usr/sbin/truescale-serdes.cmds
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Is being run?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Also what version of libipathverbs do you have?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ira
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *From:* users-bounces at lists.openfabrics.org [mailto:
>>>>>>>>> users-bounces at lists.openfabrics.org] *On Behalf Of *Weiny, Ira
>>>>>>>>> *Sent:* Wednesday, October 07, 2015 1:31 PM
>>>>>>>>> *To:* Hal Rosenstock; German Anders
>>>>>>>>>
>>>>>>>>> *Cc:* users at lists.openfabrics.org
>>>>>>>>> *Subject:* Re: [Users] IB topology config and polling state
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Agree with Hal here.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I’m not familiar with those blades/switches. I’ll ask around.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ira
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *From:* Hal Rosenstock [mailto:hal.rosenstock at gmail.com
>>>>>>>>> <hal.rosenstock at gmail.com>]
>>>>>>>>> *Sent:* Wednesday, October 07, 2015 1:26 PM
>>>>>>>>> *To:* German Anders
>>>>>>>>> *Cc:* Weiny, Ira; users at lists.openfabrics.org
>>>>>>>>> *Subject:* Re: [Users] IB topology config and polling state
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> That's the gateway to the switch in the enclosure. It's the
>>>>>>>>> internal connectivity in the blade enclosure that's (physically) broken.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Oct 7, 2015 at 4:24 PM, German Anders <
>>>>>>>>> ganders at despegar.com> wrote:
>>>>>>>>>
>>>>>>>>> cabled
>>>>>>>>>
>>>>>>>>> the blade it's:
>>>>>>>>>
>>>>>>>>> vendid=0x2c9
>>>>>>>>> devid=0xbd36
>>>>>>>>> sysimgguid=0x2c902004b0918
>>>>>>>>> switchguid=0x2c902004b0918(2c902004b0918)
>>>>>>>>> Switch 32 "S-0002c902004b0918" # "Infiniscale-IV
>>>>>>>>> Mellanox Technologies" base port 0 *lid 29* lmc 0
>>>>>>>>> [1] "S-e41d2d030031e9c1"[9] # "MF0;GWIB01:SX6036G/U1"
>>>>>>>>> lid 24 4xQDR
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *German*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2015-10-07 17:21 GMT-03:00 Hal Rosenstock <
>>>>>>>>> hal.rosenstock at gmail.com>:
>>>>>>>>>
>>>>>>>>> What are those HCAs cabled to or is it internal to the blade
>>>>>>>>> enclosure ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Oct 7, 2015 at 3:24 PM, German Anders <
>>>>>>>>> ganders at despegar.com> wrote:
>>>>>>>>>
>>>>>>>>> Yeah, is there any command that I can run in order to change the
>>>>>>>>> port state on the remote switch? I mean everything looks good but in the hp
>>>>>>>>> blades still getting:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # ibstat
>>>>>>>>> CA 'qib0'
>>>>>>>>> CA type: InfiniPath_QMH7342
>>>>>>>>> Number of ports: 2
>>>>>>>>> Firmware version:
>>>>>>>>> Hardware version: 2
>>>>>>>>> Node GUID: 0x0011750000791fec
>>>>>>>>> System image GUID: 0x0011750000791fec
>>>>>>>>> Port 1:
>>>>>>>>> State: *Down*
>>>>>>>>> Physical state: *Polling*
>>>>>>>>> Rate: 40
>>>>>>>>> Base lid: 4660
>>>>>>>>> LMC: 0
>>>>>>>>> SM lid: 4660
>>>>>>>>> Capability mask: 0x0761086a
>>>>>>>>> Port GUID: 0x0011750000791fec
>>>>>>>>> Link layer: InfiniBand
>>>>>>>>> Port 2:
>>>>>>>>> State: *Down*
>>>>>>>>> Physical state: *Polling*
>>>>>>>>> Rate: 40
>>>>>>>>> Base lid: 4660
>>>>>>>>> LMC: 0
>>>>>>>>> SM lid: 4660
>>>>>>>>> Capability mask: 0x0761086a
>>>>>>>>> Port GUID: 0x0011750000791fed
>>>>>>>>> Link layer: InfiniBand
>>>>>>>>>
>>>>>>>>> Also on working hosts I only see devices from the local network,
>>>>>>>>> but didn't see any of the blades hca connections.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *German*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2015-10-07 16:21 GMT-03:00 Hal Rosenstock <
>>>>>>>>> hal.rosenstock at gmail.com>:
>>>>>>>>>
>>>>>>>>> The screen shot looks good :-) SM brought the link up to active.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Note that the ibportstate command you gave was for switch port 0
>>>>>>>>> of the Mellanox IS-4 switch in the QLogic HP BLc 4X QDR IB Switch.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Oct 7, 2015 at 3:06 PM, German Anders <
>>>>>>>>> ganders at despegar.com> wrote:
>>>>>>>>>
>>>>>>>>> Yes, find attached an screenshot of the port information (# 9) the
>>>>>>>>> one that makes the ISL to the QLogic HP BLc 4X QDR IB Switch, also from one
>>>>>>>>> of the hosts that are connected to one of the SX6018F I can see the
>>>>>>>>> 'remote' HP IB SW:
>>>>>>>>>
>>>>>>>>> # *ibnodes*
>>>>>>>>>
>>>>>>>>> (...)
>>>>>>>>> Switch : 0x0002c902004b0918 ports 32 "Infiniscale-IV Mellanox
>>>>>>>>> Technologies" base port 0 *lid 29* lmc 0
>>>>>>>>> Switch : 0xe41d2d030031e9c1 ports 37 "MF0;GWIB01:SX6036G/U1"
>>>>>>>>> enhanced port 0 lid 24 lmc 0
>>>>>>>>> (...)
>>>>>>>>>
>>>>>>>>> # *ibportstate -L 29 query*
>>>>>>>>> Switch PortInfo:
>>>>>>>>> # Port info: Lid 29 port 0
>>>>>>>>> LinkState:.......................Active
>>>>>>>>> PhysLinkState:...................LinkUp
>>>>>>>>> Lid:.............................29
>>>>>>>>> SMLid:...........................2
>>>>>>>>> LMC:.............................0
>>>>>>>>> LinkWidthSupported:..............1X or 4X
>>>>>>>>> LinkWidthEnabled:................1X or 4X
>>>>>>>>> LinkWidthActive:.................4X
>>>>>>>>> LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
>>>>>>>>> LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
>>>>>>>>> LinkSpeedActive:.................10.0 Gbps
>>>>>>>>> Mkey:............................<not displayed>
>>>>>>>>> MkeyLeasePeriod:.................0
>>>>>>>>> ProtectBits:.....................0
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *German*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2015-10-07 16:00 GMT-03:00 Hal Rosenstock <
>>>>>>>>> hal.rosenstock at gmail.com>:
>>>>>>>>>
>>>>>>>>> One more thing hopefully before playing with the low level phy
>>>>>>>>> settings:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Are you using known good cables ? Do you have FDR cables on the
>>>>>>>>> FDR <-> FDR links ? Cable lengths can matter as well.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Oct 7, 2015 at 12:57 PM, Hal Rosenstock <
>>>>>>>>> hal.rosenstock at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Were the ports mapped to the phy profile shutdown when you changed
>>>>>>>>> this ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> LLR is a proprietary Mellanox mechanism.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> You might want 2 different profiles: one for the interfaces
>>>>>>>>> connected to other gateway interfaces (which are FDR (and FDR-10) capable
>>>>>>>>> and the other for the interfaces connecting to QDR (the older equipment in
>>>>>>>>> your network). By configuring the Switch-X interfaces to the appropriate
>>>>>>>>> possible speeds and disabling the proprietary mechanisms there, the link
>>>>>>>>> should not only come up but also this will occur faster than if FDR/FDR10
>>>>>>>>> are enabled.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I suspect that due to the Switch-X configuration that the links to
>>>>>>>>> the switch(es) in the HP enclosures do not negotiate properly (as shown by
>>>>>>>>> down rather than LinkUp).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Once you get all your links to INIT, negotiation has occurred and
>>>>>>>>> then it's time for SM to bring links to active.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Since you have down links, the SM can't do anything about those.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Oct 7, 2015 at 12:44 PM, German Anders <
>>>>>>>>> ganders at despegar.com> wrote:
>>>>>>>>>
>>>>>>>>> Anyone had any experience with HP BLc 4X QDR IB Switch?? I know
>>>>>>>>> that this kind of SW does not come with an embedded sm, but I don't know
>>>>>>>>> how to access any mgmt at all on this particularly switch, I mean for
>>>>>>>>> example to setup speed or anything like that, is possible to access through
>>>>>>>>> the chassis?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *German*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2015-10-07 13:19 GMT-03:00 German Anders <ganders at despegar.com>:
>>>>>>>>>
>>>>>>>>> I think so, but when trying to configured the phy-profile on the
>>>>>>>>> interface in order to negotiate on QDR it failed to map the profile:
>>>>>>>>>
>>>>>>>>> GWIB01 [proxy-ha-group: master] (config) # show phy-profile
>>>>>>>>> high-speed-ber
>>>>>>>>>
>>>>>>>>> Profile: high-speed-ber
>>>>>>>>> --------
>>>>>>>>> llr support ib-speed
>>>>>>>>> SDR: disable
>>>>>>>>> DDR: disable
>>>>>>>>> QDR: disable
>>>>>>>>> FDR10: enable-request
>>>>>>>>> FDR: enable-request
>>>>>>>>>
>>>>>>>>> GWIB01 [proxy-ha-group: master] (config) # show phy-profile
>>>>>>>>> hp-encl-isl
>>>>>>>>>
>>>>>>>>> Profile: hp-encl-isl
>>>>>>>>> --------
>>>>>>>>> llr support ib-speed
>>>>>>>>> SDR: disable
>>>>>>>>> DDR: disable
>>>>>>>>> QDR: enable
>>>>>>>>> FDR10: enable-request
>>>>>>>>> FDR: enable-request
>>>>>>>>>
>>>>>>>>> GWIB01 [proxy-ha-group: master] (config) #
>>>>>>>>> GWIB01 [proxy-ha-group: master] (config) # interface ib 1/9
>>>>>>>>> phy-profile map hp-encl-isl
>>>>>>>>> *% Cannot map profile hp-encl-isl to port: 1/9*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *German*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2015-10-07 13:17 GMT-03:00 Weiny, Ira <ira.weiny at intel.com>:
>>>>>>>>>
>>>>>>>>> The driver ‘qib’ is loading fine. As can be seen by the ibstat
>>>>>>>>> output. The ib_ipath is an older card.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The problem is the link is not coming up to init. Like Hal said
>>>>>>>>> the link should transition to “link up” without the SMs involvement.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I think you are on to something with the fact that it seems like
>>>>>>>>> your switch ports are not configured to do QDR.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ira
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *From:* German Anders [mailto:ganders at despegar.com]
>>>>>>>>> *Sent:* Wednesday, October 07, 2015 9:05 AM
>>>>>>>>> *To:* Weiny, Ira
>>>>>>>>> *Cc:* Hal Rosenstock; users at lists.openfabrics.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *Subject:* Re: [Users] IB topology config and polling state
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yes I've that file:
>>>>>>>>>
>>>>>>>>> /usr/sbin/truescale-serdes.cmds
>>>>>>>>>
>>>>>>>>> Also I've done the install of libipathverbs:
>>>>>>>>>
>>>>>>>>> # apt-get install libipathverbs-dev
>>>>>>>>>
>>>>>>>>> But I try to load the ib_ipath module but I'm getting the
>>>>>>>>> following error msg:
>>>>>>>>>
>>>>>>>>> # modprobe ib_ipath
>>>>>>>>> modprobe: ERROR: could not insert 'ib_ipath': Device or resource
>>>>>>>>> busy
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *German*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2015-10-07 12:54 GMT-03:00 Weiny, Ira <ira.weiny at intel.com>:
>>>>>>>>>
>>>>>>>>> There are a few issues for routing in that diagram but the links
>>>>>>>>> should come up.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I assume there is some backplane between the blade servers and the
>>>>>>>>> switch in that chassis?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Have you gotten libipathverbs installed?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> In ipathverbs there is a serdes tuning script.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://github.com/01org/libipathverbs/blob/master/truescale-serdes.cmds
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Does your libipathverbs include that file? If not try the latest
>>>>>>>>> from github.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ira
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *From:* users-bounces at lists.openfabrics.org [mailto:
>>>>>>>>> users-bounces at lists.openfabrics.org] *On Behalf Of *German Anders
>>>>>>>>> *Sent:* Wednesday, October 07, 2015 8:41 AM
>>>>>>>>> *To:* Hal Rosenstock
>>>>>>>>> *Cc:* users at lists.openfabrics.org
>>>>>>>>> *Subject:* Re: [Users] IB topology config and polling state
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Hal,
>>>>>>>>>
>>>>>>>>> Thanks for the reply, I've attach a pdf with the diagram topology,
>>>>>>>>> I don't know if this is the best way to go or if there's another way to
>>>>>>>>> connect and setup the IB network, tips and suggestions will be very
>>>>>>>>> appreciated, also the mezzanine cards are already installed on the blade
>>>>>>>>> hosts:
>>>>>>>>>
>>>>>>>>> # lspci
>>>>>>>>> (...)
>>>>>>>>> 41:00.0 InfiniBand: QLogic Corp. IBA7322 QDR InfiniBand HCA (rev
>>>>>>>>> 02)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks in advance,
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *German*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2015-10-07 11:47 GMT-03:00 Hal Rosenstock <
>>>>>>>>> hal.rosenstock at gmail.com>:
>>>>>>>>>
>>>>>>>>> Hi again German,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Looks like you made some progress from yesterday as the qib ports
>>>>>>>>> are now Polling rather than Disabled.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> But since they are Down, do you have them cabled to a switch ?
>>>>>>>>> That should bring the links up and the port state will be Init. That is the
>>>>>>>>> "starting" point.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> You will also then need to be running SM to bring the ports up to
>>>>>>>>> Active.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -- Hal
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Oct 7, 2015 at 10:37 AM, German Anders <
>>>>>>>>> ganders at despegar.com> wrote:
>>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I don't know if this is the mailist list for this kind of topic
>>>>>>>>> but I'm really new to IB and I've just install two SX6036G gateways
>>>>>>>>> connected to each other through two ISL ports, then I've configured a
>>>>>>>>> proxy-arp between both nodes (sm is disable on both gw's):
>>>>>>>>>
>>>>>>>>> GWIB01 [proxy-ha-group: master] (config) # show proxy-arp ha
>>>>>>>>>
>>>>>>>>> Load balancing algorithm: ib-base-ip
>>>>>>>>> Number of Proxy-Arp interfaces: 1
>>>>>>>>>
>>>>>>>>> Proxy-ARP VIP
>>>>>>>>> =============
>>>>>>>>> Pra-group name: proxy-ha-group
>>>>>>>>> HA VIP address: 10.xx.xx.xx/xx
>>>>>>>>>
>>>>>>>>> Active nodes:
>>>>>>>>> ID State IP
>>>>>>>>> --------------------------------------------------------------
>>>>>>>>> GWIB01 master 10.xx.xx.xx1
>>>>>>>>> GWIB02 standby 10.xx.xx.xx2
>>>>>>>>>
>>>>>>>>> Then I setup two SX6018F switches (*SWIB01* and *SWIB02*), one
>>>>>>>>> connected to GWIB01 and the other connected to GWIB02. The SM is configured
>>>>>>>>> locally on both SWIB01 & SWIB02 switches. So far so good, after this config
>>>>>>>>> I setup a commodity server with a MLNX IB ADPT FDR to the SWIB01 & SWIB02
>>>>>>>>> switches, config the drivers, etc and then get it up & running fine.
>>>>>>>>>
>>>>>>>>> Finally I've setup a HP Enclosure with an internal IB SW (then
>>>>>>>>> connect port 1 of the internal SW to GWIB01 - link is up but LLR status is
>>>>>>>>> inactive), install one of the blades and I see the following:
>>>>>>>>>
>>>>>>>>> # ibstat
>>>>>>>>> CA 'qib0'
>>>>>>>>> CA type: InfiniPath_QMH7342
>>>>>>>>> Number of ports: 2
>>>>>>>>> Firmware version:
>>>>>>>>> Hardware version: 2
>>>>>>>>> Node GUID: 0x0011750000791fec
>>>>>>>>> System image GUID: 0x0011750000791fec
>>>>>>>>> Port 1:
>>>>>>>>> State: Down
>>>>>>>>> Physical state: Polling
>>>>>>>>> Rate: 40
>>>>>>>>> Base lid: 4660
>>>>>>>>> LMC: 0
>>>>>>>>> SM lid: 4660
>>>>>>>>> Capability mask: 0x0761086a
>>>>>>>>> Port GUID: 0x0011750000791fec
>>>>>>>>> Link layer: InfiniBand
>>>>>>>>> Port 2:
>>>>>>>>> State: Down
>>>>>>>>> Physical state: Polling
>>>>>>>>> Rate: 40
>>>>>>>>> Base lid: 4660
>>>>>>>>> LMC: 0
>>>>>>>>> SM lid: 4660
>>>>>>>>> Capability mask: 0x0761086a
>>>>>>>>> Port GUID: 0x0011750000791fed
>>>>>>>>> Link layer: InfiniBand
>>>>>>>>>
>>>>>>>>> So I was wondering if maybe the SM is not being recognized on the
>>>>>>>>> Blade system and that's why is not passing the Polling state, is that
>>>>>>>>> possible? Or maybe is not possible to connect an ISL between the GW and the
>>>>>>>>> HP internal SW so that the sm is available or maybe the inactive LLR is
>>>>>>>>> causing this thing, any ideas? I thought about connecting the ISL
>>>>>>>>> of the HP IB SW to the SWIB01 or SWIB02 instead of the GW's but I don't
>>>>>>>>> have any available ports.
>>>>>>>>>
>>>>>>>>> Thanks in advance,
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *German*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Users mailing list
>>>>>>>>> Users at lists.openfabrics.org
>>>>>>>>> http://lists.openfabrics.org/mailman/listinfo/users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/users/attachments/20151013/70abb61b/attachment.html>
More information about the Users
mailing list