[Users] IB topology config and polling state
Hal Rosenstock
hal.rosenstock at gmail.com
Tue Oct 13 11:01:42 PDT 2015
SymbolErrors, PortRcvErrors, LinkErrorRecovery are all indicative of a bad
physical connection. Usually this is cables but this is a backplane
connection. Would you reseat the boards and make sure they're solidly in
their backplane connectors ?
On Tue, Oct 13, 2015 at 1:58 PM, German Anders <ganders at despegar.com> wrote:
> I see many errors increasing:
>
> # perfquery 29 28
> # Port counters: Lid 29 port 28 (CapMask: 0x1B01)
> PortSelect:......................28
> CounterSelect:...................0x0000
> SymbolErrorCounter:..............65535
> LinkErrorRecoveryCounter:........0
> LinkDownedCounter:...............0
> PortRcvErrors:...................20904
> PortRcvRemotePhysicalErrors:.....0
> PortRcvSwitchRelayErrors:........0
> PortXmitDiscards:................0
> PortXmitConstraintErrors:........0
> PortRcvConstraintErrors:.........0
> CounterSelect2:..................0x00
> LocalLinkIntegrityErrors:........0
> ExcessiveBufferOverrunErrors:....0
> VL15Dropped:.....................0
> PortXmitData:....................166
> PortRcvData:.....................167
> PortXmitPkts:....................3
> PortRcvPkts:.....................3
> PortXmitWait:....................0
>
> # perfquery 29 28
> # Port counters: Lid 29 port 28 (CapMask: 0x1B01)
> PortSelect:......................28
> CounterSelect:...................0x0000
> SymbolErrorCounter:..............65535
> LinkErrorRecoveryCounter:........4
> LinkDownedCounter:...............0
> PortRcvErrors:...................37474
> PortRcvRemotePhysicalErrors:.....0
> PortRcvSwitchRelayErrors:........0
> PortXmitDiscards:................0
> PortXmitConstraintErrors:........0
> PortRcvConstraintErrors:.........0
> CounterSelect2:..................0x00
> LocalLinkIntegrityErrors:........0
> ExcessiveBufferOverrunErrors:....0
> VL15Dropped:.....................0
> PortXmitData:....................260
> PortRcvData:.....................262
> PortXmitPkts:....................5
> PortRcvPkts:.....................5
> PortXmitWait:....................0
>
> # perfquery 29 28
> # Port counters: Lid 29 port 28 (CapMask: 0x1B01)
> PortSelect:......................28
> CounterSelect:...................0x0000
> SymbolErrorCounter:..............65535
> LinkErrorRecoveryCounter:........4
> LinkDownedCounter:...............0
> PortRcvErrors:...................60240
> PortRcvRemotePhysicalErrors:.....0
> PortRcvSwitchRelayErrors:........0
> PortXmitDiscards:................0
> PortXmitConstraintErrors:........0
> PortRcvConstraintErrors:.........0
> CounterSelect2:..................0x00
> LocalLinkIntegrityErrors:........0
> ExcessiveBufferOverrunErrors:....0
> VL15Dropped:.....................0
> PortXmitData:....................449
> PortRcvData:.....................452
> PortXmitPkts:....................9
> PortRcvPkts:.....................9
> PortXmitWait:....................0
>
> *after changing the speed port to 7:*
>
> # perfquery 29 28
> # Port counters: Lid 29 port 28 (CapMask: 0x1B01)
> PortSelect:......................28
> CounterSelect:...................0x0000
> SymbolErrorCounter:..............65535
> LinkErrorRecoveryCounter:........2
> LinkDownedCounter:...............0
> PortRcvErrors:...................1568
> PortRcvRemotePhysicalErrors:.....0
> PortRcvSwitchRelayErrors:........0
> PortXmitDiscards:................0
> PortXmitConstraintErrors:........0
> PortRcvConstraintErrors:.........0
> CounterSelect2:..................0x00
> LocalLinkIntegrityErrors:........0
> ExcessiveBufferOverrunErrors:....0
> VL15Dropped:.....................0
> PortXmitData:....................0
> PortRcvData:.....................0
> PortXmitPkts:....................0
> PortRcvPkts:.....................0
> PortXmitWait:....................0
>
> # perfquery 29 28
> # Port counters: Lid 29 port 28 (CapMask: 0x1B01)
> PortSelect:......................28
> CounterSelect:...................0x0000
> SymbolErrorCounter:..............65535
> LinkErrorRecoveryCounter:........2
> LinkDownedCounter:...............0
> PortRcvErrors:...................11356
> PortRcvRemotePhysicalErrors:.....0
> PortRcvSwitchRelayErrors:........0
> PortXmitDiscards:................0
> PortXmitConstraintErrors:........0
> PortRcvConstraintErrors:.........0
> CounterSelect2:..................0x00
> LocalLinkIntegrityErrors:........0
> ExcessiveBufferOverrunErrors:....0
> VL15Dropped:.....................0
> PortXmitData:....................166
> PortRcvData:.....................167
> PortXmitPkts:....................3
> PortRcvPkts:.....................3
> PortXmitWait:....................0
>
> # perfquery 29 28
> # Port counters: Lid 29 port 28 (CapMask: 0x1B01)
> PortSelect:......................28
> CounterSelect:...................0x0000
> SymbolErrorCounter:..............65535
> LinkErrorRecoveryCounter:........3
> LinkDownedCounter:...............0
> PortRcvErrors:...................44675
> PortRcvRemotePhysicalErrors:.....0
> PortRcvSwitchRelayErrors:........0
> PortXmitDiscards:................0
> PortXmitConstraintErrors:........0
> PortRcvConstraintErrors:.........0
> CounterSelect2:..................0x00
> LocalLinkIntegrityErrors:........0
> ExcessiveBufferOverrunErrors:....0
> VL15Dropped:.....................0
> PortXmitData:....................261
> PortRcvData:.....................284
> PortXmitPkts:....................5
> PortRcvPkts:.....................6
> PortXmitWait:....................0
>
>
>
> *German* <ganders at despegar.com>
>
> 2015-10-13 14:26 GMT-03:00 Hal Rosenstock <hal.rosenstock at gmail.com>:
>
>> Speed 4 is not valid as an enabled speed. It probably is speed 7 (SDR,
>> DDR, or QDR) . Since qib is QDR (only), espeed shouldn't be used. QDR only
>> is non standard which is why it says "IBA extension".
>>
>> As to why the link comes up and then goes down, there are a number of
>> possible reasons. What does perfquery 29 28 say ? Are there some error
>> counters increasing there ? If so, which ones ?
>>
>> On Tue, Oct 13, 2015 at 1:09 PM, German Anders <ganders at despegar.com>
>> wrote:
>>
>>> Ok, some good news! I've find the way to get LinkUp on the blade port...
>>> but.. not for so long :S, it seems that after making the following changes
>>> the ports at the end come up...but then in a couple of minutes goes down
>>> again:
>>>
>>> - Move the Encl IB SW to Bay7
>>> - Move the Blade Mezz cards to slot 3
>>>
>>> Then for ex:
>>>
>>> Blade IB SW - LID: 29
>>> Blade 4 IB-SW-PORT 28
>>>
>>> 1)
>>>
>>> # ibportstate -L 29 28
>>> Switch PortInfo:
>>> # Port info: Lid 29 port 28
>>> LinkState:.......................Down
>>> PhysLinkState:...................Polling
>>> Lid:.............................75
>>> SMLid:...........................2328
>>> LMC:.............................0
>>> LinkWidthSupported:..............1X or 4X
>>> LinkWidthEnabled:................1X or 4X
>>> LinkWidthActive:.................4X
>>> LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
>>> LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
>>> LinkSpeedActive:.................2.5 Gbps
>>>
>>>
>>> 2)
>>>
>>> # ibportstate -L 29 28 disable
>>> # ibportstate -L 29 28 speed 4
>>> # ibportstate -L 29 28 espeed 4
>>> # ibportstate -L 29 28 smlid 2
>>> # ibportstate -L 29 28 enable
>>>
>>> 3)
>>>
>>> # ibportstate -L 29 28
>>> Switch PortInfo:
>>> # Port info: Lid 29 port 28
>>> LinkState:.......................Active
>>> PhysLinkState:...................LinkUp
>>> Lid:.............................75
>>> SMLid:...........................2328
>>> LMC:.............................0
>>> LinkWidthSupported:..............1X or 4X
>>> LinkWidthEnabled:................1X or 4X
>>> LinkWidthActive:.................4X
>>> LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
>>> LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
>>> LinkSpeedActive:.................10.0 Gbps
>>> Peer PortInfo:
>>> # Port info: Lid 29 DR path slid 4; dlid 65535; 0,28 port 1
>>> LinkState:.......................Active
>>> PhysLinkState:...................LinkUp
>>> Lid:.............................30
>>> SMLid:...........................2
>>> LMC:.............................0
>>> LinkWidthSupported:..............1X or 4X
>>> LinkWidthEnabled:................1X or 4X
>>> LinkWidthActive:.................4X
>>> LinkSpeedSupported:..............10.0 Gbps (IBA extension)
>>> LinkSpeedEnabled:................10.0 Gbps (IBA extension)
>>> LinkSpeedActive:.................10.0 Gbps
>>> Mkey:............................<not displayed>
>>> MkeyLeasePeriod:.................0
>>> ProtectBits:.....................0
>>>
>>>
>>> # ibstat
>>> CA 'qib0'
>>> CA type: InfiniPath_QMH7342
>>> Number of ports: 2
>>> Firmware version:
>>> Hardware version: 2
>>> Node GUID: 0x0011750000791fec
>>> System image GUID: 0x0011750000791fec
>>> Port 1:
>>> State: *Active*
>>> Physical state: *LinkUp*
>>> Rate: 40
>>> Base lid: 30
>>> LMC: 0
>>> SM lid: 2
>>> Capability mask: 0x0761086a
>>> Port GUID: 0x0011750000791fec
>>> Link layer: InfiniBand
>>> Port 2:
>>> State: Down
>>> Physical state: Polling
>>> Rate: 40
>>> Base lid: 65535
>>> LMC: 0
>>> SM lid: 65535
>>> Capability mask: 0x0761086a
>>> Port GUID: 0x0011750000791fed
>>> Link layer: InfiniBand
>>>
>>>
>>> a couple of minutes then...
>>>
>>> # ibportstate -L 29 28
>>> Switch PortInfo:
>>> # Port info: Lid 29 port 28
>>> LinkState:.......................Down
>>> PhysLinkState:...................Polling
>>> Lid:.............................75
>>> SMLid:...........................2328
>>> LMC:.............................0
>>> LinkWidthSupported:..............1X or 4X
>>> LinkWidthEnabled:................1X or 4X
>>> LinkWidthActive:.................4X
>>> LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps
>>> LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps
>>> LinkSpeedActive:.................2.5 Gbps
>>>
>>>
>>> *German* <ganders at despegar.com>
>>>
>>> 2015-10-13 12:02 GMT-03:00 German Anders <ganders at despegar.com>:
>>>
>>>> Can anyone that's using the HP BLc 4X QDR IB Switch tell me in what
>>>> interconnect bay is configured the switch? I've that in bay3 & mezz 1, I'll
>>>> try to move to bay7 & mezz 3 and see if at least something changed or not,
>>>> any useful info will be really appreciated.
>>>>
>>>> Cheers,
>>>>
>>>>
>>>> *German* <ganders at despegar.com>
>>>>
>>>> 2015-10-13 10:51 GMT-03:00 German Anders <ganders at despegar.com>:
>>>>
>>>>> yes, but the switch does not come with an external interface... just
>>>>> the i2c port :S
>>>>>
>>>>>
>>>>>
>>>>> *German* <ganders at despegar.com>
>>>>>
>>>>> 2015-10-13 10:49 GMT-03:00 Hal Rosenstock <hal.rosenstock at gmail.com>:
>>>>>
>>>>>> I don't really know but I was wondering about whether your hardware
>>>>>> supports this.
>>>>>>
>>>>>> HPOA_2 shows internal interface to OA is absent as is external
>>>>>> Ethernet interface.
>>>>>>
>>>>>> On Tue, Oct 13, 2015 at 9:31 AM, German Anders <ganders at despegar.com>
>>>>>> wrote:
>>>>>>
>>>>>>> when trying to assign the ip addr to the interconnect bay it seems
>>>>>>> that it does not 'apply' the change, and the ip addr is not displayed as
>>>>>>> 'configured' so there's no way to enter from outside. ideas?
>>>>>>>
>>>>>>> *German*
>>>>>>>
>>>>>>> 2015-10-13 10:15 GMT-03:00 Hal Rosenstock <hal.rosenstock at gmail.com>
>>>>>>> :
>>>>>>>
>>>>>>>> What about the 3 critical errors ? What are they ?
>>>>>>>>
>>>>>>>> On Tue, Oct 13, 2015 at 9:13 AM, German Anders <
>>>>>>>> ganders at despegar.com> wrote:
>>>>>>>>
>>>>>>>>> I've try that and in fact I try to put the IP ADDR like this:
>>>>>>>>>
>>>>>>>>> SET EBIPA INTERCONNECT XX.XX.XX.XX XXX.XXX.XXX.XXX 3
>>>>>>>>> SET EBIPA INTERCONNECT GATEWAY XX.XX.XX.XX 3
>>>>>>>>> SET EBIPA INTERCONNECT DOMAIN "xxxxxx.net" 3
>>>>>>>>> ADD EBIPA INTERCONNECT DNS 10.xx.xx.xx 3
>>>>>>>>> ADD EBIPA INTERCONNECT DNS 10.xx.xx.xx 3
>>>>>>>>> SET EBIPA INTERCONNECT NTP PRIMARY NONE 3
>>>>>>>>> SET EBIPA INTERCONNECT NTP SECONDARY NONE 3
>>>>>>>>> ENABLE EBIPA INTERCONNECT 3
>>>>>>>>>
>>>>>>>>> SAVE EBIPA
>>>>>>>>>
>>>>>>>>> But i'm not getting any ip response, also I've try many diff ip addr with no luck...if i put that ip to one of the blades it works fine, but not to the interconnect bay :( any other idea?
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *German* <ganders at despegar.com>
>>>>>>>>>
>>>>>>>>> 2015-10-13 10:01 GMT-03:00 Hal Rosenstock <
>>>>>>>>> hal.rosenstock at gmail.com>:
>>>>>>>>>
>>>>>>>>>> Looks like there are 3 critical errors in system status. Did you
>>>>>>>>>> look at these ?
>>>>>>>>>>
>>>>>>>>>> I don't know if you've seen this but there is some info on
>>>>>>>>>> configuring the management IPs in
>>>>>>>>>> http://h10032.www1.hp.com/ctg/Manual/c00814176.pdf
>>>>>>>>>>
>>>>>>>>>> Have you looked at/tried the command line interface ?
>>>>>>>>>>
>>>>>>>>>> On Tue, Oct 13, 2015 at 8:28 AM, German Anders <
>>>>>>>>>> ganders at despegar.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Hal,
>>>>>>>>>>>
>>>>>>>>>>> It does not allow me to setup an IP ADDR to the Internal SW so I
>>>>>>>>>>> can't access from outside, except from the tools that I mentioned before,
>>>>>>>>>>> also it doesn't allow me to access through serial connection from inside
>>>>>>>>>>> the enclosure. I've attach some screen-shots about the connectivity.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *German*
>>>>>>>>>>>
>>>>>>>>>>> 2015-10-13 9:13 GMT-03:00 Hal Rosenstock <
>>>>>>>>>>> hal.rosenstock at gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>>> Hi German,
>>>>>>>>>>>>
>>>>>>>>>>>> Are the cards in the correct bays and slots ?
>>>>>>>>>>>>
>>>>>>>>>>>> Do you have the HP Onboard Administrator tool ? What does it
>>>>>>>>>>>> say about internal connectivity ?
>>>>>>>>>>>>
>>>>>>>>>>>> -- Hal
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Oct 13, 2015 at 7:44 AM, German Anders <
>>>>>>>>>>>> ganders at despegar.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Ira,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've some HP documentation but it quite short, also it doesn't
>>>>>>>>>>>>> describe any 'config' or 'impl' steps in order to get the internal switch
>>>>>>>>>>>>> up and running. The version of SW that came with enclosure does not had any
>>>>>>>>>>>>> management module at all, so it depends on external management. During
>>>>>>>>>>>>> weekend I've found a way to upgrade the firmware of the HP switch with the
>>>>>>>>>>>>> following command (mlxburn -d lid-0x001D -fw fw-IS4.mlx), and the I've run
>>>>>>>>>>>>> (flint -d /dev/mst/SW_MT48438_0x2c902004b0918_lid-0x001D dc
>>>>>>>>>>>>> /home/ceph/HPIBSW.INI) and found the following inside that file:
>>>>>>>>>>>>>
>>>>>>>>>>>>> [PS_INFO]
>>>>>>>>>>>>> Name = 489184-B21
>>>>>>>>>>>>> Description = HP BLc 4X QDR IB Switch
>>>>>>>>>>>>>
>>>>>>>>>>>>> [ADAPTER]
>>>>>>>>>>>>> PSID = HP_0100000009
>>>>>>>>>>>>>
>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>
>>>>>>>>>>>>> [IB_TO_HW_MAP]
>>>>>>>>>>>>> PORT1=14
>>>>>>>>>>>>> PORT2=15
>>>>>>>>>>>>> PORT3=16
>>>>>>>>>>>>> PORT4=17
>>>>>>>>>>>>> PORT5=18
>>>>>>>>>>>>> PORT6=12
>>>>>>>>>>>>> PORT7=11
>>>>>>>>>>>>> PORT8=10
>>>>>>>>>>>>> PORT9=9
>>>>>>>>>>>>> PORT10=8
>>>>>>>>>>>>> PORT11=7
>>>>>>>>>>>>> PORT12=6
>>>>>>>>>>>>> PORT13=5
>>>>>>>>>>>>> PORT14=4
>>>>>>>>>>>>> PORT15=3
>>>>>>>>>>>>> PORT16=2
>>>>>>>>>>>>> PORT17=20
>>>>>>>>>>>>> PORT18=22
>>>>>>>>>>>>> PORT19=24
>>>>>>>>>>>>>
>>>>>>>>>>>>> PORT20=26
>>>>>>>>>>>>> PORT21=28
>>>>>>>>>>>>> PORT22=30
>>>>>>>>>>>>> PORT23=35
>>>>>>>>>>>>> PORT24=33
>>>>>>>>>>>>> PORT25=21
>>>>>>>>>>>>> PORT26=23
>>>>>>>>>>>>> PORT27=25
>>>>>>>>>>>>> PORT28=27
>>>>>>>>>>>>> PORT29=29
>>>>>>>>>>>>> PORT30=36
>>>>>>>>>>>>> PORT31=34
>>>>>>>>>>>>> PORT32=32
>>>>>>>>>>>>> PORT33=1
>>>>>>>>>>>>> PORT34=13
>>>>>>>>>>>>> PORT35=19
>>>>>>>>>>>>> PORT36=31
>>>>>>>>>>>>>
>>>>>>>>>>>>> [unused_ports]
>>>>>>>>>>>>> hw_port1_not_in_use=1
>>>>>>>>>>>>> hw_port13_not_in_use=1
>>>>>>>>>>>>> hw_port19_not_in_use=1
>>>>>>>>>>>>> hw_port31_not_in_use=1
>>>>>>>>>>>>>
>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>
>>>>>>>>>>>>> I don't know if maybe there's some issue with the port
>>>>>>>>>>>>> mapping, anyone had used this kind of switch?
>>>>>>>>>>>>>
>>>>>>>>>>>>> The summary of the problem is correct, the connectivity
>>>>>>>>>>>>> between the IB network (MLNX switches/gw) and the HP IB switch is working
>>>>>>>>>>>>> since I was able to upgrade the firmare of the switch and get information
>>>>>>>>>>>>> about it. But, the connection between the mezzanine cards of the blades and
>>>>>>>>>>>>> the internal IB sw enclosure is not working at all. Note, that if I go to
>>>>>>>>>>>>> the OA administration of the enclosure I can see the 'green' ports mapping
>>>>>>>>>>>>> of each of the blades and the interconnection switch, so I'm guessing that
>>>>>>>>>>>>> it should be working.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regarding the questions:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1) What type of switch is in the HP chassis?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> *QLogic HP BLc 4X QDR IB Switch*
>>>>>>>>>>>>>
>>>>>>>>>>>>> *PSID = HP_0100000009*
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Image type: FS2*
>>>>>>>>>>>>>
>>>>>>>>>>>>> *FW ver: 7.4.3000*
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Device ID: 48438*
>>>>>>>>>>>>> *GUI: 0002c902004b0918*
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2) Do you have console access or http access to that
>>>>>>>>>>>>> switch?
>>>>>>>>>>>>>
>>>>>>>>>>>>> *No, since it didn't had any manage module mezzanine card
>>>>>>>>>>>>> inside the switch, it only come with a i2c port. But, i can have access
>>>>>>>>>>>>> through the mlxburn and flint tools from one host that's connected to the
>>>>>>>>>>>>> ib network (outside the enclosure).*
>>>>>>>>>>>>>
>>>>>>>>>>>>> 3) Does that switch have an SM in it?
>>>>>>>>>>>>>
>>>>>>>>>>>>> *No*
>>>>>>>>>>>>>
>>>>>>>>>>>>> 4) What version of the kernel are you running with the
>>>>>>>>>>>>> qib cards?
>>>>>>>>>>>>>
>>>>>>>>>>>>> a. I assume you are using the qib driver in that kernel.
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Ubuntu 14.04.3 LTS - kernel 3.18.20-031820-generic*
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> At some point Hal spoke of “LLR being a Mellanox thing” Was
>>>>>>>>>>>>> that to solve the problem of connecting the “HP switch” to the Mellanox
>>>>>>>>>>>>> switch?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> *No, since LLR is only supported between mlnx devices, the ISL
>>>>>>>>>>>>> are up and working, since it's possible for me to query the switch*
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I would like it if you could verify that the
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> /usr/sbin/truescale-serdes.cmds
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is being run?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> *When trying to run the command:*
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> *#
>>>>>>>>>>>>> /usr/sbin/truescale-serdes.cmds/usr/sbin/truescale-serdes.cmds: 100:
>>>>>>>>>>>>> /usr/sbin/truescale-serdes.cmds: Syntax error: "(" unexpected (expecting
>>>>>>>>>>>>> "}")*
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also what version of libipathverbs do you have?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> *# rpm -qa | grep libipathverbslibipathverbs-1.3-1.x86_64*
>>>>>>>>>>>>> Thanks in advance,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> *German*
>>>>>>>>>>>>> 2015-10-13 2:14 GMT-03:00 Weiny, Ira <ira.weiny at intel.com>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> German,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Do you have any documentation on the HP blade system? And
>>>>>>>>>>>>>> the switch which is in that system?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have to admit I have not followed everything in this thread
>>>>>>>>>>>>>> regarding your configuration but it seems like you have some mellanox
>>>>>>>>>>>>>> switches connected into an HP chassis which has both a switch and blades
>>>>>>>>>>>>>> with qib (Truescale) cards.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The connection from the mellanox switch to the “HP chassis
>>>>>>>>>>>>>> switch” is linkup (active) but the connections to the individual qib HCAs
>>>>>>>>>>>>>> are not even linkup.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is that a correct summary of the problem?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If so here are some questions:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1) What type of switch is in the HP chassis?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2) Do you have console access or http access to that
>>>>>>>>>>>>>> switch?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 3) Does that switch have an SM in it?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 4) What version of the kernel are you running with the
>>>>>>>>>>>>>> qib cards?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> a. I assume you are using the qib driver in that
>>>>>>>>>>>>>> kernel.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> At some point Hal spoke of “LLR being a Mellanox thing” Was
>>>>>>>>>>>>>> that to solve the problem of connecting the “HP switch” to the Mellanox
>>>>>>>>>>>>>> switch?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I would like it if you could verify that the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> /usr/sbin/truescale-serdes.cmds
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is being run?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also what version of libipathverbs do you have?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ira
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *From:* users-bounces at lists.openfabrics.org [mailto:
>>>>>>>>>>>>>> users-bounces at lists.openfabrics.org] *On Behalf Of *Weiny,
>>>>>>>>>>>>>> Ira
>>>>>>>>>>>>>> *Sent:* Wednesday, October 07, 2015 1:31 PM
>>>>>>>>>>>>>> *To:* Hal Rosenstock; German Anders
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Cc:* users at lists.openfabrics.org
>>>>>>>>>>>>>> *Subject:* Re: [Users] IB topology config and polling state
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Agree with Hal here.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I’m not familiar with those blades/switches. I’ll ask around.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ira
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *From:* Hal Rosenstock [mailto:hal.rosenstock at gmail.com
>>>>>>>>>>>>>> <hal.rosenstock at gmail.com>]
>>>>>>>>>>>>>> *Sent:* Wednesday, October 07, 2015 1:26 PM
>>>>>>>>>>>>>> *To:* German Anders
>>>>>>>>>>>>>> *Cc:* Weiny, Ira; users at lists.openfabrics.org
>>>>>>>>>>>>>> *Subject:* Re: [Users] IB topology config and polling state
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That's the gateway to the switch in the enclosure. It's the
>>>>>>>>>>>>>> internal connectivity in the blade enclosure that's (physically) broken.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Oct 7, 2015 at 4:24 PM, German Anders <
>>>>>>>>>>>>>> ganders at despegar.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> cabled
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> the blade it's:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> vendid=0x2c9
>>>>>>>>>>>>>> devid=0xbd36
>>>>>>>>>>>>>> sysimgguid=0x2c902004b0918
>>>>>>>>>>>>>> switchguid=0x2c902004b0918(2c902004b0918)
>>>>>>>>>>>>>> Switch 32 "S-0002c902004b0918" # "Infiniscale-IV
>>>>>>>>>>>>>> Mellanox Technologies" base port 0 *lid 29* lmc 0
>>>>>>>>>>>>>> [1] "S-e41d2d030031e9c1"[9] #
>>>>>>>>>>>>>> "MF0;GWIB01:SX6036G/U1" lid 24 4xQDR
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *German*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2015-10-07 17:21 GMT-03:00 Hal Rosenstock <
>>>>>>>>>>>>>> hal.rosenstock at gmail.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What are those HCAs cabled to or is it internal to the blade
>>>>>>>>>>>>>> enclosure ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Oct 7, 2015 at 3:24 PM, German Anders <
>>>>>>>>>>>>>> ganders at despegar.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yeah, is there any command that I can run in order to change
>>>>>>>>>>>>>> the port state on the remote switch? I mean everything looks good but in
>>>>>>>>>>>>>> the hp blades still getting:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> # ibstat
>>>>>>>>>>>>>> CA 'qib0'
>>>>>>>>>>>>>> CA type: InfiniPath_QMH7342
>>>>>>>>>>>>>> Number of ports: 2
>>>>>>>>>>>>>> Firmware version:
>>>>>>>>>>>>>> Hardware version: 2
>>>>>>>>>>>>>> Node GUID: 0x0011750000791fec
>>>>>>>>>>>>>> System image GUID: 0x0011750000791fec
>>>>>>>>>>>>>> Port 1:
>>>>>>>>>>>>>> State: *Down*
>>>>>>>>>>>>>> Physical state: *Polling*
>>>>>>>>>>>>>> Rate: 40
>>>>>>>>>>>>>> Base lid: 4660
>>>>>>>>>>>>>> LMC: 0
>>>>>>>>>>>>>> SM lid: 4660
>>>>>>>>>>>>>> Capability mask: 0x0761086a
>>>>>>>>>>>>>> Port GUID: 0x0011750000791fec
>>>>>>>>>>>>>> Link layer: InfiniBand
>>>>>>>>>>>>>> Port 2:
>>>>>>>>>>>>>> State: *Down*
>>>>>>>>>>>>>> Physical state: *Polling*
>>>>>>>>>>>>>> Rate: 40
>>>>>>>>>>>>>> Base lid: 4660
>>>>>>>>>>>>>> LMC: 0
>>>>>>>>>>>>>> SM lid: 4660
>>>>>>>>>>>>>> Capability mask: 0x0761086a
>>>>>>>>>>>>>> Port GUID: 0x0011750000791fed
>>>>>>>>>>>>>> Link layer: InfiniBand
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also on working hosts I only see devices from the local
>>>>>>>>>>>>>> network, but didn't see any of the blades hca connections.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *German*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2015-10-07 16:21 GMT-03:00 Hal Rosenstock <
>>>>>>>>>>>>>> hal.rosenstock at gmail.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The screen shot looks good :-) SM brought the link up to
>>>>>>>>>>>>>> active.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Note that the ibportstate command you gave was for switch
>>>>>>>>>>>>>> port 0 of the Mellanox IS-4 switch in the QLogic HP BLc 4X QDR IB Switch.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Oct 7, 2015 at 3:06 PM, German Anders <
>>>>>>>>>>>>>> ganders at despegar.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, find attached an screenshot of the port information (#
>>>>>>>>>>>>>> 9) the one that makes the ISL to the QLogic HP BLc 4X QDR IB Switch, also
>>>>>>>>>>>>>> from one of the hosts that are connected to one of the SX6018F I can see
>>>>>>>>>>>>>> the 'remote' HP IB SW:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> # *ibnodes*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>> Switch : 0x0002c902004b0918 ports 32 "Infiniscale-IV
>>>>>>>>>>>>>> Mellanox Technologies" base port 0 *lid 29* lmc 0
>>>>>>>>>>>>>> Switch : 0xe41d2d030031e9c1 ports 37
>>>>>>>>>>>>>> "MF0;GWIB01:SX6036G/U1" enhanced port 0 lid 24 lmc 0
>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> # *ibportstate -L 29 query*
>>>>>>>>>>>>>> Switch PortInfo:
>>>>>>>>>>>>>> # Port info: Lid 29 port 0
>>>>>>>>>>>>>> LinkState:.......................Active
>>>>>>>>>>>>>> PhysLinkState:...................LinkUp
>>>>>>>>>>>>>> Lid:.............................29
>>>>>>>>>>>>>> SMLid:...........................2
>>>>>>>>>>>>>> LMC:.............................0
>>>>>>>>>>>>>> LinkWidthSupported:..............1X or 4X
>>>>>>>>>>>>>> LinkWidthEnabled:................1X or 4X
>>>>>>>>>>>>>> LinkWidthActive:.................4X
>>>>>>>>>>>>>> LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0
>>>>>>>>>>>>>> Gbps
>>>>>>>>>>>>>> LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0
>>>>>>>>>>>>>> Gbps
>>>>>>>>>>>>>> LinkSpeedActive:.................10.0 Gbps
>>>>>>>>>>>>>> Mkey:............................<not displayed>
>>>>>>>>>>>>>> MkeyLeasePeriod:.................0
>>>>>>>>>>>>>> ProtectBits:.....................0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *German*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2015-10-07 16:00 GMT-03:00 Hal Rosenstock <
>>>>>>>>>>>>>> hal.rosenstock at gmail.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> One more thing hopefully before playing with the low level
>>>>>>>>>>>>>> phy settings:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Are you using known good cables ? Do you have FDR cables on
>>>>>>>>>>>>>> the FDR <-> FDR links ? Cable lengths can matter as well.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Oct 7, 2015 at 12:57 PM, Hal Rosenstock <
>>>>>>>>>>>>>> hal.rosenstock at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Were the ports mapped to the phy profile shutdown when you
>>>>>>>>>>>>>> changed this ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> LLR is a proprietary Mellanox mechanism.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You might want 2 different profiles: one for the interfaces
>>>>>>>>>>>>>> connected to other gateway interfaces (which are FDR (and FDR-10) capable
>>>>>>>>>>>>>> and the other for the interfaces connecting to QDR (the older equipment in
>>>>>>>>>>>>>> your network). By configuring the Switch-X interfaces to the appropriate
>>>>>>>>>>>>>> possible speeds and disabling the proprietary mechanisms there, the link
>>>>>>>>>>>>>> should not only come up but also this will occur faster than if FDR/FDR10
>>>>>>>>>>>>>> are enabled.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I suspect that due to the Switch-X configuration that the
>>>>>>>>>>>>>> links to the switch(es) in the HP enclosures do not negotiate properly (as
>>>>>>>>>>>>>> shown by down rather than LinkUp).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Once you get all your links to INIT, negotiation has occurred
>>>>>>>>>>>>>> and then it's time for SM to bring links to active.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Since you have down links, the SM can't do anything about
>>>>>>>>>>>>>> those.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Oct 7, 2015 at 12:44 PM, German Anders <
>>>>>>>>>>>>>> ganders at despegar.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Anyone had any experience with HP BLc 4X QDR IB Switch?? I
>>>>>>>>>>>>>> know that this kind of SW does not come with an embedded sm, but I don't
>>>>>>>>>>>>>> know how to access any mgmt at all on this particularly switch, I mean for
>>>>>>>>>>>>>> example to setup speed or anything like that, is possible to access through
>>>>>>>>>>>>>> the chassis?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *German*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2015-10-07 13:19 GMT-03:00 German Anders <
>>>>>>>>>>>>>> ganders at despegar.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think so, but when trying to configured the phy-profile on
>>>>>>>>>>>>>> the interface in order to negotiate on QDR it failed to map the profile:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> GWIB01 [proxy-ha-group: master] (config) # show phy-profile
>>>>>>>>>>>>>> high-speed-ber
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Profile: high-speed-ber
>>>>>>>>>>>>>> --------
>>>>>>>>>>>>>> llr support ib-speed
>>>>>>>>>>>>>> SDR: disable
>>>>>>>>>>>>>> DDR: disable
>>>>>>>>>>>>>> QDR: disable
>>>>>>>>>>>>>> FDR10: enable-request
>>>>>>>>>>>>>> FDR: enable-request
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> GWIB01 [proxy-ha-group: master] (config) # show phy-profile
>>>>>>>>>>>>>> hp-encl-isl
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Profile: hp-encl-isl
>>>>>>>>>>>>>> --------
>>>>>>>>>>>>>> llr support ib-speed
>>>>>>>>>>>>>> SDR: disable
>>>>>>>>>>>>>> DDR: disable
>>>>>>>>>>>>>> QDR: enable
>>>>>>>>>>>>>> FDR10: enable-request
>>>>>>>>>>>>>> FDR: enable-request
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> GWIB01 [proxy-ha-group: master] (config) #
>>>>>>>>>>>>>> GWIB01 [proxy-ha-group: master] (config) # interface ib 1/9
>>>>>>>>>>>>>> phy-profile map hp-encl-isl
>>>>>>>>>>>>>> *% Cannot map profile hp-encl-isl to port: 1/9*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *German*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2015-10-07 13:17 GMT-03:00 Weiny, Ira <ira.weiny at intel.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The driver ‘qib’ is loading fine. As can be seen by the
>>>>>>>>>>>>>> ibstat output. The ib_ipath is an older card.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The problem is the link is not coming up to init. Like Hal
>>>>>>>>>>>>>> said the link should transition to “link up” without the SMs involvement.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think you are on to something with the fact that it seems
>>>>>>>>>>>>>> like your switch ports are not configured to do QDR.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ira
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *From:* German Anders [mailto:ganders at despegar.com]
>>>>>>>>>>>>>> *Sent:* Wednesday, October 07, 2015 9:05 AM
>>>>>>>>>>>>>> *To:* Weiny, Ira
>>>>>>>>>>>>>> *Cc:* Hal Rosenstock; users at lists.openfabrics.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *Subject:* Re: [Users] IB topology config and polling state
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes I've that file:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> /usr/sbin/truescale-serdes.cmds
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also I've done the install of libipathverbs:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> # apt-get install libipathverbs-dev
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> But I try to load the ib_ipath module but I'm getting the
>>>>>>>>>>>>>> following error msg:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> # modprobe ib_ipath
>>>>>>>>>>>>>> modprobe: ERROR: could not insert 'ib_ipath': Device or
>>>>>>>>>>>>>> resource busy
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *German*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2015-10-07 12:54 GMT-03:00 Weiny, Ira <ira.weiny at intel.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> There are a few issues for routing in that diagram but the
>>>>>>>>>>>>>> links should come up.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I assume there is some backplane between the blade servers
>>>>>>>>>>>>>> and the switch in that chassis?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Have you gotten libipathverbs installed?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In ipathverbs there is a serdes tuning script.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://github.com/01org/libipathverbs/blob/master/truescale-serdes.cmds
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Does your libipathverbs include that file? If not try the
>>>>>>>>>>>>>> latest from github.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ira
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *From:* users-bounces at lists.openfabrics.org [mailto:
>>>>>>>>>>>>>> users-bounces at lists.openfabrics.org] *On Behalf Of *German
>>>>>>>>>>>>>> Anders
>>>>>>>>>>>>>> *Sent:* Wednesday, October 07, 2015 8:41 AM
>>>>>>>>>>>>>> *To:* Hal Rosenstock
>>>>>>>>>>>>>> *Cc:* users at lists.openfabrics.org
>>>>>>>>>>>>>> *Subject:* Re: [Users] IB topology config and polling state
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Hal,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the reply, I've attach a pdf with the diagram
>>>>>>>>>>>>>> topology, I don't know if this is the best way to go or if there's another
>>>>>>>>>>>>>> way to connect and setup the IB network, tips and suggestions will be very
>>>>>>>>>>>>>> appreciated, also the mezzanine cards are already installed on the blade
>>>>>>>>>>>>>> hosts:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> # lspci
>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>> 41:00.0 InfiniBand: QLogic Corp. IBA7322 QDR InfiniBand HCA
>>>>>>>>>>>>>> (rev 02)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks in advance,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *German*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2015-10-07 11:47 GMT-03:00 Hal Rosenstock <
>>>>>>>>>>>>>> hal.rosenstock at gmail.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi again German,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Looks like you made some progress from yesterday as the qib
>>>>>>>>>>>>>> ports are now Polling rather than Disabled.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> But since they are Down, do you have them cabled to a switch
>>>>>>>>>>>>>> ? That should bring the links up and the port state will be Init. That is
>>>>>>>>>>>>>> the "starting" point.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You will also then need to be running SM to bring the ports
>>>>>>>>>>>>>> up to Active.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -- Hal
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Oct 7, 2015 at 10:37 AM, German Anders <
>>>>>>>>>>>>>> ganders at despegar.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I don't know if this is the mailist list for this kind of
>>>>>>>>>>>>>> topic but I'm really new to IB and I've just install two SX6036G gateways
>>>>>>>>>>>>>> connected to each other through two ISL ports, then I've configured a
>>>>>>>>>>>>>> proxy-arp between both nodes (sm is disable on both gw's):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> GWIB01 [proxy-ha-group: master] (config) # show proxy-arp ha
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Load balancing algorithm: ib-base-ip
>>>>>>>>>>>>>> Number of Proxy-Arp interfaces: 1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Proxy-ARP VIP
>>>>>>>>>>>>>> =============
>>>>>>>>>>>>>> Pra-group name: proxy-ha-group
>>>>>>>>>>>>>> HA VIP address: 10.xx.xx.xx/xx
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Active nodes:
>>>>>>>>>>>>>> ID State IP
>>>>>>>>>>>>>> --------------------------------------------------------------
>>>>>>>>>>>>>> GWIB01 master 10.xx.xx.xx1
>>>>>>>>>>>>>> GWIB02 standby 10.xx.xx.xx2
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Then I setup two SX6018F switches (*SWIB01* and *SWIB02*),
>>>>>>>>>>>>>> one connected to GWIB01 and the other connected to GWIB02. The SM is
>>>>>>>>>>>>>> configured locally on both SWIB01 & SWIB02 switches. So far so good, after
>>>>>>>>>>>>>> this config I setup a commodity server with a MLNX IB ADPT FDR to the
>>>>>>>>>>>>>> SWIB01 & SWIB02 switches, config the drivers, etc and then get it up &
>>>>>>>>>>>>>> running fine.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Finally I've setup a HP Enclosure with an internal IB SW
>>>>>>>>>>>>>> (then connect port 1 of the internal SW to GWIB01 - link is up but LLR
>>>>>>>>>>>>>> status is inactive), install one of the blades and I see the following:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> # ibstat
>>>>>>>>>>>>>> CA 'qib0'
>>>>>>>>>>>>>> CA type: InfiniPath_QMH7342
>>>>>>>>>>>>>> Number of ports: 2
>>>>>>>>>>>>>> Firmware version:
>>>>>>>>>>>>>> Hardware version: 2
>>>>>>>>>>>>>> Node GUID: 0x0011750000791fec
>>>>>>>>>>>>>> System image GUID: 0x0011750000791fec
>>>>>>>>>>>>>> Port 1:
>>>>>>>>>>>>>> State: Down
>>>>>>>>>>>>>> Physical state: Polling
>>>>>>>>>>>>>> Rate: 40
>>>>>>>>>>>>>> Base lid: 4660
>>>>>>>>>>>>>> LMC: 0
>>>>>>>>>>>>>> SM lid: 4660
>>>>>>>>>>>>>> Capability mask: 0x0761086a
>>>>>>>>>>>>>> Port GUID: 0x0011750000791fec
>>>>>>>>>>>>>> Link layer: InfiniBand
>>>>>>>>>>>>>> Port 2:
>>>>>>>>>>>>>> State: Down
>>>>>>>>>>>>>> Physical state: Polling
>>>>>>>>>>>>>> Rate: 40
>>>>>>>>>>>>>> Base lid: 4660
>>>>>>>>>>>>>> LMC: 0
>>>>>>>>>>>>>> SM lid: 4660
>>>>>>>>>>>>>> Capability mask: 0x0761086a
>>>>>>>>>>>>>> Port GUID: 0x0011750000791fed
>>>>>>>>>>>>>> Link layer: InfiniBand
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So I was wondering if maybe the SM is not being recognized on
>>>>>>>>>>>>>> the Blade system and that's why is not passing the Polling state, is that
>>>>>>>>>>>>>> possible? Or maybe is not possible to connect an ISL between the GW and the
>>>>>>>>>>>>>> HP internal SW so that the sm is available or maybe the inactive LLR is
>>>>>>>>>>>>>> causing this thing, any ideas? I thought about connecting
>>>>>>>>>>>>>> the ISL of the HP IB SW to the SWIB01 or SWIB02 instead of the GW's but I
>>>>>>>>>>>>>> don't have any available ports.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks in advance,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *German*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Users mailing list
>>>>>>>>>>>>>> Users at lists.openfabrics.org
>>>>>>>>>>>>>> http://lists.openfabrics.org/mailman/listinfo/users
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/users/attachments/20151013/a9adabee/attachment.html>
More information about the Users
mailing list