[ewg] (no subject)

Sun Oct 18 09:15:06 PDT 2009

On Sat, Oct 17, 2009 at 2:49 PM, Mahmoud Hanafi <mhanafi at csc.com> wrote:
> Hal Rosenstock <hal.rosenstock at gmail.com> wrote on 10/17/2009 04:34:55 AM:
>
>> Hal Rosenstock <hal.rosenstock at gmail.com>
>> 10/17/2009 04:34 AM
>>
>> To
>>
>> Mahmoud Hanafi/DEF/CSC at CSC
>>
>> cc
>>
>> ewg at lists.openfabrics.org
>>
>> Subject
>>
>> Re: [ewg] (no subject)
>>
>> On Fri, Oct 16, 2009 at 2:20 PM, Mahmoud Hanafi <mhanafi at csc.com> wrote:
>> > We have a linux cluster running RH5.3 with ofed1.4 using Mellanox
>> > MT25418.
>> > The cluster is attached to a sun solaris10.7 thumper box. The thumper
>> > box
>> > export a zfs filesystem via NFS. linux clients mount the filesystem via
>> > IPoIB.
>> >
>> > Under filesystem I/O load the subnet manager gets repeated path record
>> > requests from the sun solaris box.
>>
>> Do the path records all look the same or different in terms of
>> destinations (and sources) ?
>
> There are path records look-ups for all the nodes repeating. So what I have
> posted below repeats in addition to all the other nodes.
>
>>
>> Is the source GUID (0x0003ba000100d0a5) the Solaris thumper port GUID
>> (00-03-BA (hex) Sun Microsystems Inc.) ? The destination appears to be
>> some HP device (00-23-7D (hex) Hewlett Packard).
>
> That is correct 0x0003ba000100d0a5 is the Solaris thumper the destination
> are HP blades See ibnetdiscover output below
>
>>
>> > This can bring the SM and the fabric down.
>>
>> Are you referring to the load due to path requests or something else ?
>
> The sun ib interface does so many path record lookup that it can't service
> nfs request. This cause the nodes to lockup.

It looked like the right status if using Get or 1 record if GetTable
is returned at least in the OpenSM case. Not sure why the thumper
continues to rerequest the same path records.

> In addition the openSM see a
> high load trying to service the requests. We where running the sm on the
> switch it was seeing a load of >3. That's when we moved it to the head node.
> It uses >%30 of a core when servicing the requests.

This is "normal" in that it takes CPU to respond to the SA queries for
PathRecords to the thumper. The amount of CPU is dependent on the
request load.

>> Running the OpenSM are the logging level you appear to be using would
>> certainly slow things down greatly so I presume that was only done to
>> look further into what was going on.
>>
>> > Any any one else had issue with solaris IB <-> Linux IB?
>>
>> I haven't run Solaris <-> Linux IB in several years now but this used
>> to work but there have been a lot of changes.
>>
>> > Any insight into what could be causing the issue?
>>
>> Could you elaborate on the below ? I see one PathRecord response trace
>> and an ibdiagnet run which shows a bad link at direct route 1,11,23
>> from where that was run. You might want to debug the issue with that
>> link.
> We have a second cluster that does the same thing. It doesn't have any bad
> links.
>
> What does the IPoIB subnet check warning mean?

-I- Subnet: IPv4 PKey:Ox7fff QKey:Ox00000blb MTU:2048Byte rate:lOGbps
SL:OxOO
-W- Suboptimal rate for group. Lowest member rate:20Gbps > grouprate:
lOGbps

It means you should be able to increase the rate of your IPoIB broadcast group.

-- Hal

>>
>> -- Hal
>>
>> >
>> > Thanks,
>> > Mahmoud
>> >
>> > ----
>> >
>> > Oct 15 19:37:
>> >
>> > 59 952368 [41E02960] 0x08 -> PathRecord dump:
>> >
>> > service id ..............0x0000000000000000
>> >
>> > dgid .................... Oxfe80000000000000 : 0x00237dffff949819
>> >
>> > sgid .................... Oxfe80000000000000 : 0x0003ba000100d0a5
>> >
>> > dlid .................... 0
>> >
>> > slid .................... 0
>> >
>> > hop_flow_raw............ OxO
>> >
>> > tclass .................. OxO
>> >
>> > num_path_revers......... Ox81
>> >
>> > pkey .................... 0x0
>> >
>> > qos_class ............... OxO
>> >
>> > sl ......................OxO
>> >
>> > mtu .....................OxO
>> >
>> > rate .................... OxO
>> >
>> > pkt_life ................0x0
>> >
>> > preference .............. 0x0
>> >
>> > resv2 ................... OxO
>> >
>> > resv3 ................... OxO
>> >
>> > Oct 15 19:37:59 952376 [41E02960) 0x08 -> osm_pr_rcv_process: Unicast
>> >
>> > destination requested
>> >
>> > Oct 15 19:37:59 952382 [41E02960] 0x08 ->
>> >
>> > osm_pr_rcv_get_port pair_paths: Src port 0x0003ba000100d0a5, Dst port
>> >
>> > 0x00237dffff949819
>> >
>> > Oct 15 19:37:59 952388 [41E02960] 0x08 ->
>> >
>> > _osm_pr_rcv_get_port_pair_paths: Src LIDs [2 - 2], Dest LIDs [67-67]
>> >
>> > Oct 15 19:37:59 952393 [41E02960] 0x08 ->
>> >
>> > _osm pr_rcv_get_lid_pair_path: Src LID 2, Dest LID 67
>> >
>> > Oct 15 19:37:59 952399 [41E02960] 0x08 -> _osm_pr_rcv_get-path_parms:
>> >
>> > Path min MTU = 4, min rate = 6
>> >
>> > Oct 15 19:37:59 952408 [41E02960] 0x08 - > _osm_pr_rcv_get-path_parms:
>> >
>> > Path params: mtu = 4, rate = 6, packet lifetime = 18, pkey = OxFFFF, sl
>> >
>> > = 0
>> >
>> > Oct 15 19:37:59 952417 [41E02960] 0x08 - > _osm_pr_rcv_get_path_parms:
>> >
>> > Path min MTU = 4, min rate = 6
>> >
>> > Oct 15 19: 37:59 952423 [41E02960] 0x08 -> osm pr_rcv_get_path parms:
>> >
>> > Path params: mtu = 4, rate
>> >
>> > = 6, packet lifetime = 18, pkey = OxFFFF, sl
>> >
>> > = 0
>> >
>> > Oct 15 19:37:59 952428 [41E02960] 0x08
>> >
>> > -> osm_sa_respond: Returning 1
>> >
>> > records
>> >
>> > Oct 15 19:37:59 952433 [41E02960] 0x08 -
>> >
>> >> osm_vendor_get: Acquiring UMAD
>> >
>> > for p_madw = 0x2a9567f2c8, size = 120
>> >
>> > Oct 15 19:37:59 952439 [41E02960] 0x08 -> osm_vendor_get: Acquired UMAD
>> >
>> > 0x2a9567f390, size = 120
>> >
>> > Oct 15 19:37:59 952455 [41E02960] 0x08 -
>> >
>> >> osm_vendor_put: Retiring UMAD
>> >
>> > 0x2a9567f390
>> >
>> > Oct 15 19:37:59 952460 [41E02960] 0x08 ->
>> >
>> > •.osm_vendor_send: Completed
>> >
>> > sending response or unsolicited p_madw'"j= Ox2a9567f2b0
>> >
>> > Oct 15 19:37:59 952466 [41E02960] 0x08 -> osm
>> >
>> > _vendor_put: Retiring UMAD
>> >
>> > 0x724520
>> >
>> > ===============
>> >
>> > Loading IBDIAGNET from: /usr/1ib64
>> >
>> > / ibdiagnetl.2
>> >
>> > -W- Topology file is not specified.
>> >
>> > Reports regarding cluster links will use direct routes.
>> >
>> > Loading IBDM from: /usr/lib64 / ibdml.2
>> >
>> > - I- Using port 1 as the local port.
>> >
>> > - I- Discovering ... 103 nodes (7 Switches & 96 CA- s) discovered.
>> >
>> > -I ---------------------------------------------------
>> >
>> > - I- Bad Guids /LIDs Info
>> >
>> > -I -------------------------------------------------- -
>> >
>> > -I- No bad Guids were found
>> >
>> > -I -------------------------------------------------- -
>> >
>> > -I- Links With Logical State = INIT
>> >
>> > -I -------------------------------------------------- -
>> >
>> > -I- No bad Links (with logical state
>> >
>> > = INIT) were found
>> >
>> > -I ---------------------------------------------------
>> >
>> > -I- PM Counters Info
>> >
>> > -I -------------------------------------------------- -
>> >
>> > -I- No illegal PM counters values were found
>> >
>> > -I ---------------------------------------------------
>> >
>> > -I- Fabric Partitions Report (see ibdiagnet.pkey for a full hosts list)
>> >
>> > -I ---------------------------------------------------
>> >
>> > -I- PKey:Ox7fff Hosts:97 full:97 partial:0
>> >
>> > -I -------------------------------------------------- -
>> >
>> > -I- IPoIB Subnets Check
>> >
>> > -I ---------------------------------------------------
>> >
>> > -I- Subnet: IPv4 PKey:Ox7fff QKey:Ox00000blb MTU:2048Byte rate:lOGbps
>> >
>> > SL:OxOO
>> >
>> > -W- Suboptimal rate for group. Lowest member rate:20Gbps > grouprate:
>> >
>> > lOGbps
>> >
>> > -I ---------------------------------------------------
>> >
>> > -I- Bad Links Info
>> >
>> > -I- Errors have occurred on the following links
>> >
>> > (for errors details, look in log file
>> >
>> > / tmp/ibdiagnet.log):
>> >
>> > -I ----------------------------------------------------
>> >
>> > Link at the end of direct route "1,11,23"
>> >
>> > ----------------------------------------------------------------
>> >
>> > -I- Stages Status Report:
>> >
>> > STAGE
>> >
>> > Bad GUIDs
>> >
>> > /LIDS Check
>> >
>> > Link State Active Check
>> >
>> > Performance Counters Report
>> >
>> > Partitions Check
>> >
>> > IPoIB Subnets Check
>> >
>> > Link Errors Check
>> >
>> > Errors Warnings
>> >
>> > 0 0
>> >
>> > 0 0
>> >
>> > 0 0
>> >
>> > 0 0
>> >
>> > 0 1
>> >
>> > 0 0
>> >
>> > Please see
>> >
>> > /tmp/ibdiagnet.log for complete log
>> >
>> > - I- Done. Run time was 6 seconds.
>> >
>> > _______________________________________________
>> > ewg mailing list
>> > ewg at lists.openfabrics.org
>> > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
>> >
>
> # Topology file: generated on Thu Oct 15 18:29:22 2009
> # Max of 3 hops discovered
> # Initiated from node 00leObffff4cO824 port 00leObffff4cO825
>
> vendid=Ox8f1
> devid=0x5a30
> sysimgguid=0x8f1040041811b
> switchguid=0x8f10400418l1a(8fl040041811a)
> Switch 24 "S-0008f1040041811a" # "ISR9024D Voltaire" base port 0 lid 6 lmc 0
> [23] "S-0019bbfffff8eab7"[1] # "MT47396 Infiniscale-III Mellanox
> Technologies" lid 7 4xDDR
> [22] IS-0019bbfffff8eade"[1] # "MT47396 Infiniscale-III Mellanox
> Technologies" lid 9 4xDDR
> [21] "S-00237dffff9427b3"[1] # "MT47396 Infiniscale-III Mellanox
> Technologies" lid 55 4xDDR
> [14] "H-0003ba000lOOdOa4"[1](3ba000lOOdOa5) # "Sun Microsystems InfiniBand
> Eagle DDR HCA" lid 2 4xDDR
> [13] "H-0003ba000100d0a4"[2](3ba000lOOdOa6) # "Sun Microsystems InfiniBand
> Eagle DDR HCA" lid 3 4xDDR
> [ll] "S-0019bbfffff8eac7"[1] # "MT47396 Infiniscale-III Mellanox
> Technologies" lid 8 4xDDR
> [10] "S-0019bbfffff8fac7"[1] # "MT47396 Infiniscale-III Mellanox
> Technologies" lid 10 4xDDR
> [9] "S-00237dffff9427ae"[1] # "MT47396 Infiniscale-III Mellanox
> Technologies" lid 54 4xDDR
> [12] "H-00leObffff4cO824"[1](leObffff4cO825) # "head1 HCA-1" lid 12 4xDDR
>
> vendid=0x2c9
> devid=Oxb924
> sysimgguid=Oxl9bbfffff8eab7
> switchguid=Oxl9bbfffff8eab7(19bbfffff8eab7)
> Switch 24 "S-0019bbfffff8eab7" # "MT47396 Infiniscale-III Mellanox
> Technologies" base port 0 lid 7 lmc 0
> [24] "H-00leobffff84bec8"[1](leobffff84bec9) # "gn115 HCA-1" lid 35 4xDDR
> [23] "H-00237dffff94f8b4"[1](237dffff94f8b5) # "gn114 HCA-1" lid 98 4xDDR
> [22] "H-00leobffff84aea4"[1](leobffff84aea5) # "gn113 HCA-1" lid 110 4xDDR
> [21] "H-00leObffff849d9c"[1](leobffff849d9d) # "gn112 HCA-1" lid 15 4xDDR
> [20] "H-00237dffff94f8f8" [1] (237dffff94f8f9) # "gnlll HCA-1" lid 103 4xDDR
> [19] "H-00237dffff94e82c"[1](237dffff94e82d) # "gn110 HCA-1" lid 97 4xDDR
> [18] "H-00leobffff84bebc"[1](leobffff84bebd) # "gn109 HCA-1" lid 33 4xDDR
> [17] "H-00237dffff94f8eO"[1](237dffff94f8el) # "gn108 HCA-1" lid 99 4xDDR
> [16] "H-00237dffff94f8ec"[1](237dffff94f8ed) # "gn107 HCA-1" lid 100 4xDDR
> [15] "H-00237dffff94f8fO"[1](237dffff94f8fl) # "gn106 HCA-1" lid 101 4xDDR
> [14] "H-00leObffff842c84"[1](le0bffff8t^c85) # "gn105 HCA-1" lid 13 4xDDR
> [13] "H-00leObffff84cel4"[1](leobffff84cel5) # "gn104 HCA-1" lid 38 4xDDR
> [12] "H-00237dffff94f8f4"[11(237dffff94f8f5) # "gn103 HCA-1" lid 102 4xDDR
> [11] "H-00leObffff849d90"[1](leobffff849d9l) # "gn102 HCA-1" lid 14 4xDDR
> [10] "H-00le0bffff84bed4"[1](leobffff84bed5) # "gn101 HCA-1" lid 36 4xDDR
> [9] "H-00237dffff94e8lc"[1](237dffff94e8ld) # "gn100 HCA-1" lid 96 4xDDR
> [1] "S-0008f1040041811a"[23] # "ISR9024D Voltaire" lid 6 4xDDR
> vendid=0x2c9
> devid= 0xb924
> sysimgguid=Oxl9bbfffff8eade
> switchguid= Oxl9bbfffff8eade(19bbfffff8eade)
> Switch 24 "S-0019bbfffff8eade" # "MT47396 Infiniscale-III Mellanox
> Technologies" base port 0 lid 9 lmc 0
> [24] "H-00237dffff94d834"[1](237dffff94d835) # "gn131 HCA-1" lid 92 4xDDR
> [23] "H-00237dffff94d824"[1](237dffff94d825) # "gn130 HCA-1" lid 88 4xDDR
> [22] "H-00237dffff94d818"[1](237dffff94d819) # "gn129 HCA-1" lid 86 4xDDR
> [21] "H-00leobffff84be6c"[1](leobffff84be6d) # "gn128 HCA-1" lid 20 4xDDR
> [20] "H-00leObffff84be78"[1](leobffff84be79) # "gn127 HCA-1" lid 109 4xDDR
> [19] "H-00leobffff84bea0"[11(leobffff84beal) # "gn126 HCA-1" lid 31 4xDDR
> [18] "H-00leobffff84be94"[l](leobffff84be95) # "gn125 HCA-1" lid 28 4xDDR
> [17] "H-00leObffff84be74"[11(leobffff84be75) # "gn124 HCA-1" lid 21 4xDDR
> [16] "H-00237dffff94d82c"[1](237dffff94d82d) # "gn123 HCA-1" lid 90 4xDDR
> [15] "H-00237dffff94c8fc"[1](237dffff94c8fd) # "gn122 HCA-1" lid 81 4xDDR
> [14] "H-00237dffff94d83c"[1](237dffff94d83d) # "gn121 HCA-1" lid 93 4xDDR
> [13] "H-00237dffff94d830"[l](237dffff94d831) # "gn120 HCA-1" lid 91 4xDDR
> [12] "H-00237dffff94d820"[1](237dffff94d821) # "gn119 HCA-1" lid 87 4xDDR
> [11] "H-00237dffff94d8Oc"[1](237dffff94d80d) # "gn118 HCA-1" lid 83 4xDDR
> [10] "H-00leobffff84be80"[1](leobffff84be8l) # "gn117 HCA-1" lid 23 4xDDR
> [9] "H-00leobffff84beb4"[1](leobffff84beb5) # "gn116 HCA-1" lid 32 4xDDR
> [1] "S-0008flO40041811a"[22] # "ISR9024D Voltaire" lid 6 4xDDR
>
> vendid=0x2c9
> devid= 0xb924
> sysimgguid=0x237dffff9427b3
> switchguid= 0x237dffff9427b3(237dffff9427b3)
> Switch 24 "S-00237dffff9427b3" ,# "MT47396 Infiniscale-III Mellanox
> Technologies" base port 0 lids lmc 0
> [24] "H-00leobffff84eclc"[1](leobffff84ecld) # "gn195 HCA-1" lid 41 4xDDR
> [23] "H-00237dffff949918"[1](237dffff949919) # "gn194 HCA-1" lid 71 4xDDR
> [22] "H-00leobffff84ccd0"[l](leobffff84ccdl) # "gn193 HCA-1" lid 37 4xDDR
> [21] "H-00237dffff944930"[1](237dffff944931) # "gn192 HCA-1" lid 62 4xDDR
> [20] "H-001eObffff84ec44"[1](leObffff84ec45) # "gnl9l HCA-1" lid 44 4xDDR
> [19] "H-00237dffff944938"[1](237dffff944939) # "gnl90 HCA-1" lid 63 4xDDR
> [18] "H-00237dffff949904"[1](237dffff949905) # "gn189 HCA-1" lid 70 4xDDR
> [17] "H-00leObffff84ec48"[1](leobffff84ec49) # "gn188 HCA-1" lid 45 4xDDR
> [16] "H-00237dffff9429d8"[1](237dffff9429d9) # "gn187 HCA-1" lid 56 4xDDR
> [15] "H-00leobffff84ec34"[1](le0bffff84ec35) # "gn186 HCA-1" lid 42 4xDDR
> [14] "H-00237dffff941b90"[1](237dffff94lb9l) # "gnl85 HCA-1" lid 49 4xDDR
> [13] "H-00leObffff84ec38"[1](leobffff84ec39) # "gnl84 HCA-1" lid 43 4xDDR
> [12] "H-00237dffff94lb8c"[1](237dffff94lb8d) # "gnl83 HCA-1" lid 48 4xDDR
> [11] "H-00237dffff941b88"[1](237dffff94lb89) # "gnl82 HCA-1" lid 47 4xDDR
> [10] "H-00leObffff84ec64"[1](leobffff84ec65) # "gnl81 HCA-1" lid 46 4xDDR
> [9] "H-00237dffff941ba8"[1](237dffff941ba9) # "gnl80 HCA-1" lid 50 4xDDR
> [1] "S-0008f1040041811a"[21] # "ISR9024D Voltaire" lid 6 4xDDR
>
> vendid=0x2c9
> devid=0xb924
> sysimgguid= Oxl9bbfffff8eac7
> switchguid=Oxl9bbfffff8eac7(19bbfffff8eac7)
> Switch 24 "S-0019bbfffff8eac7" # "MT47396 Infiniscale-III
> Mellanox Technologies" base port 0 lid 8 lmc 0
> [24] "H-00leobffff84aed4"[1](leobffff84aed5) # "gnl63 HCA-1" lid 18 4xDDR
> [22] "H-00leobffff84be9c"[1](leobffff84be9d) # "gnl6l HCA-1" lid 30 4xDDR
> [21] "H-00237dffff949828"[1](237dffff949829) # "gnl60 HCA-1" lid 69 4xDDR
> [20] "H-00237dffff94c854"[1](237dffff94c855) # "gn159 HCA-1" lid 76 4xDDR
> [19] "H-00237dffff94d804"[1](237dffff94d805) # "gnl58 HCA-1" lid 82 4xDDR
> [18] "H-00237dffff94981c"[1](237dffff94981d) # "gn157 HCA-1" lid 68 4xDDR
> [17] "H-00237dffff949800"[1](237dffff949801) # "gnl56 HCA-1" lid 66 4xDDR
> [16] "H-00leObffff84bec4"[1)(leobffff84bec5) # "gnl55 HCA-1" lid 34 4xDDR
> [15] "H-00leobffff84aec4"[1](leobffff84aec5) # "gnl54 HCA-1" lid 17 4xDDR
> [14] "H-00leobffff84aeac"[1](le0bffff8 ead) # "gnl53 HCA-1" lid 16 4xDDR
> [13] "H-00237dffff949818"[11(237dffff949819) # "gnl52 HCA-1" lid 67 4xDDR
> [12] "H-00leobffff84aee0"[1](leobffff84aeel) # "gnl51 HCA-1" lid 19 4xDDR
> [11] "H-00leObffff84ce90"[1](leobffff84ce9l) # "gnl50 HCA-1" lid 39 4xDDR
> [10] "H-00237dffff94b8lc"[1](237dffff94b8ld) # "gn149 HCA-1" lid 75 4xDDR
> [9] "H-00237dffff94a8fc"[1](237dffff94a8fd) # "gn148 HCA-1" lid 74 4xDDR
> [1] "S-0008f1040041811a"[11] # "ISR9024D Voltaire" lid 6 4xDDR
>
> vendid=0x2c9
> devid= 0xb924
> sysimgguid= Oxl9bbfffff8fac7
> switchguid= Oxl9bbfffff8fac7(19bbfffff8fac7)
> Switch 24 "S-0019bbfffff8fac7" # "MT47396 Infiniscale-III Mellanox
> Technologies" base port 0 lid 10 lmc 0
> [24] "H-00237dffff94c8f4"[1](237dffff94c8f5) # "gn147 HCA-1" lid 79 4xDDR
> [23] "H-00237dffff94c8f8"[1](237dffff94c8f9) # "gn146 HCA-1" lid 80 4xDDR
> [22] "H-00237dffff94c8fO"[1](237dffff94c8fl) # "gn145 HCA-1" lid 78 4xDDR
> [21] "H-00237dffff94d844"[1](237dffff94d845) # "gnl44 HCA-1" lid 94 4xDDR
> [19] "H-00leObffff84be98"[1](leObffff84be99) # "gnl42 HCA-1" lid 29 4xDDR
> [18] "H-001eobffff84be7c"[1](leobffff84be7d) # "gnl4l HCA-1" lid 22 4xDDR
> [17] "H-00le0bffff84be84"[1](leobffff84be85) # "gnl40 HCA-1" lid 24 4xDDR
> (16] "H-00237dffff94d828"[1](237dffff94d829) # "gn139 HCA-1" lid 89 4xDDR
> [15] "H-00237dffff94d8lO"[1](237dffff94d8ll) # "gn138 HCA-1" lid 84 4xDDR
> [14] "H-00237dffff94d814"[1](237dffff94d815) # "gnl37 HCA-1" lid 85 4xDDR
> [13] "H-00237dffff94c8ec"[1](237dffff94c8ed) # "gn136 HCA-1" lid 77 4xDDR
> (12] "H-00237dffff94d848"[1](237dffff94d849) # "gn135 HCA-1" lid 95 4xDDR
> [11] "H-00237dffff94a8e4"[1](237dffff94a8e5) # "gn134 HCA-1" lid 73 4xDDR
> [10] "H-00leobffff84be88"[1](leobffff84be89) # "gnl33 HCA-1" lid 25 4xDDR
> [9] "H-00leobffff84be8c"[1](leobffff84be8d) # "gn132 HCA-1" lid 26 4xDDR
> [1] "S-0008flO4004l8lla"[10] # "ISR9024D Voltaire" lid 6 4xDDR
>
> vendid=0x2c9
> devid= 0xb924
> sysimgguid= 0x237dffff9427ae
> switchguid= Ox237dffff9427ae(237dffff9427ae)
> Switch 24 "S-00237dffff9427ae" # "MT47396 Infiniscale-III Mellanox
> Technologies" base port 0 lid 54 lmc 0
> [24] "H-00237dffff943984"[1](237dffff943985) # "gn179 HCA-1" lid 57 4xDDR
> [23] "H-00237dffff949924"[1](237dffff94 925) # "gn178 HCA-1" lid 72 4xDDR
> [22] "H-00237dffff944940"[1](237dffff944941) # "gn177 HCA-1" lid 64 4xDDR
> [21] "H-00237dffff943994"[1](237dffff943995) # "gn176 HCA-1" lid 59 4xDDR
> [20] "H-00237dffff94ldbO"[1](237dffff94ldbl) # "gn175 HCA-1" lid 53 4xDDR
> (19] "H-00237dffff94fac8"[1](237dffff94fac9) # "gn174 HCA-1" lid 105 4xDDR
> [18] "H-00237dffff944944"[1](237dffff944945) # "gn173 HCA-1" lid 65 4xDDR
> [17] "H-00237dffff94399c"[1](237dffff94399d) # "gn172 HCA-1" lid 61 4xDDR
> [16] "H-00237dffff941da0"[1](237dffff941dal) # "gnl71 HCA-1" lid 52 4xDDR
> [15] "H-00237dffff943998"[1](237dffff943999) # "gnl70 HCA-1" lid 60 4xDDR
> [14] "H-00237dffff94facc"[1](237dffff94facd) # "gnl69 HCA-1" lid 106 4xDDR
> [13] "H-00237dffff943990"[1](237dffff943991) # "gnl68 HCA-1" lid 58 4xDDR
> [12] "H-00237dffff941bcc"[1](237dffff941bcd) # "gn167 HCA-1" lid 51 4xDDR
> [11] "H-00237dffff94fad0"[1](237dffff94fad1) # "gnl66 HCA-1" lid 107 4xDDR
> [10] "H-00237dffff94fad4"[1](237dffff94fad5) # "gn165 HCA-1" lid 108 4xDDR
> [9] "H-00237dffff94facO"[1](237dffff94facl) # "gnl64 HCA-1" lid 104 4xDDR
> [l] "S-0008flO40041811a"[9] # "ISR9024D Voltaire" lid 6 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid=0x237dffff943987
> caguid=0x237dffff943984
> Ca 2 "H - 00237dffff943984" # "gnl79 HCA-1"
> [1](237dffff943985) "S-00237dffff9427ae"[24] # lid 57 lmc 0 "MT47396
> Infiniscale - III Mellanox Technologies" lid 54 4xDDR
>
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff949927
> caguid= 0x237dffff949924
> Ca 2 "H - 00237dffff949924" # "gnl78 HCA-1"
> [1](237dffff949925) "S-00237dffff9427ae"[23] # lid 72 lmc 0 "MT47396
> Infiniscale - III Mellanox Technologies" lid 54 4xDDR
>
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff944943
> caguid=0x237dffff944940
> Ca 2 "H-00237dffff944940" # "gn177 HCA-1"
> [1](237dffff944941) "S-00237dffff9427ae"[22] # lid 64 lmc 0 "MT47396
> Infiniscale - III Mellanox Technologies" lid 54 4xDDR
>
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= 0x237dffff943997
> caguid= 0x237dffff943994
> Ca 2 "H- 00237dffff943994" # 11 9,A1,176 HCA-1"
> [1] (237dffff943995) "S-00237dffff94' ae" [21] # lid 59 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 54 4xDDR
>
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff94ldb3
> caguid=Ox237dffff94ldbO
> Ca 2 "H-00237dffff941dbO" # "gn175 HCA-1"
> [1](237dffff94ldb1) "S-00237dffff9427ae"[20] # lid 53 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 54 4xDDR
>
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff 94facb
> caguid=Ox237dffff94fac8
> Ca 2 "H-00237dffff94fac8" # "gn174 HCA-1"
> [1](237dffff94fac9) "S-00237dffff9427ae"[19] # lid 105
> lmc 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 54 4xDDR
>
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff944947
> caguid=Ox237dffff944944
> Ca 2 "H-00237dffff944944" # "gn173 HCA-1"
> [1](237dffff944945) "S-00237dffff9427ae"[18] # lid 65 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 54 4xDDR
>
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= 0x237dffff94399f
> caguid=0x237dffff94399c
> Ca 2 "H-00237dffff94399c" # "gn172 HCA-1"
> [1](237dffff94399d) "S-00237dffff9427ae"[17] # lid 61 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 54 4xDDR
>
>
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= 0x237dffff941da3
> caguid=0x237dffff941da0
> Ca 2 "H-00237dffff941da0" # "gn171 HCA-1"
> [1](237dffff941dal) "S-00237dffff9427ae"[16] # lid 52 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 54 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= 0x237dffff94399b
> caguid= 0x237dffff943998
> Ca 2 "H- 00237dffff943998" # "gn170 HCA-1"
> [1](237dffff943999) "S-00237dffff9427ae"[15] # lid 60 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 54 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff94facf
> caguid=0x237dffff94facc
> Ca 2 "H-00237dffff94facc" # "gn169 HCA-1"
> [1](237dffff94facd) "S-00237dffff9427ae"[14] # lid 106
> lmc 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 54 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= 0x237dffff943993
> caguid=0x237dffff943990
> Ca 2 "H-00237dffff943990" # "gn168 HCA-1"
> [1](237dffff943991) "S-00237dffff9427ae"[13] # lid 58 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 54 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid=0x237dffff941bcf
> caguid=0x237dffff941bcc
> Ca 2 "H-00237dffff941bcc" # "gn167 HCA-1"
> [1](237dffff941bcd) "S-00237dffff9427ae"[12] # lid 51 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 54 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= 0x237dffff94fad3
> caguid=Ox237dffff94fad0
> Ca 2 "H-00237dffff94fadO" # "gn166 HCA-1"
> [1] (237dffff94fadl) "S-00237dffff9427ae" [11] # lid 107
> lmc 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 54 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid=0x237dffff94fad7
> caguid=0x237dffff94fad4
> Ca 2 "H - 00237dffff94fad4" # "gn165 HCA-1"
> [1](237dffff94fad5) "S-00237dffff9427ae"[10] # lid 108
> lmc 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 54 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= 0x237dffff94fac3
> caguid=0x237dffff94fac0
> Ca 2 "H - 00237dffff94facO" # "gn164 HCA-1"
> [1](237dffff94facl) "S-00237dffff9427ae"[9] # lid 104 lmc 0
> "MT47396 Infiniscale - III Mellanox Technologies" lid 54 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= Ox237dffff94c8f7
> caguid=Ox237dffff94c8f4
> Ca 2 "H-00237dffff94c8f4" # "gn147 HCA-1"
> [1] (237dffff94c8f5) "S-0019bbfffff8fac7" [24] # lid 79 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 10 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= 0x237dffff94c8fb
> caguid=Ox237dffff94c8f8
> Ca 2 "H-00237dffff94c8f8" # "gn146 HCA-1"
> [1] (237dffff94c8f9) "S-0019bbfffff8fac7" [23] # lid 80 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 10 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= Ox237dffff94c8f3
> caguid=Ox237dffff94c8f0
> Ca 2 "H- 00237dffff94c8fO" # "gn145 HCA-1"
> [1](237dffff94c8fl) "S-0019bbfffff8fac7"[22] # lid 78 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 10 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff94d847
> caguid=0x237dffff94d844
> Ca 2 "H - 00237dffff94d844" # "gn144 HCA-1"
> [1] (237dffff94d845) "S-0019bbfffff8fac7" [21] # lid 94 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 10 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= OxleObffff84be9b
> caguid=Oxleobffff84be98
> Ca 2 "H - 001eObffff84be98" # "gn142 HCA-1"
> [1](leobffff84be99) "S-0019bbfffff8fac7"[19] # lid 29 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 10 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= OxleObffff84be7f
> caguid=Oxleobffff84be7c
> Ca 2 "H-001eObffff84be7c" # "gn141 HCA-1"
> [1](leobffff84be7d) "S-0019bbfffff8fac7"[18] # lid 22 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 10 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= OxleObffff84be87
> caguid=OxleObffff84be84
> Ca 2 "H-001eObffff84be84" # "gn140 HCA-1"
> [1](leobffff84be85) "S-0019bbfffff8fac7"[17] # lid 24 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 10 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= 0x237dffff94d82b
> caguid= 0x237dffff94d828
> Ca 2 "H- 00237dffff94d828" # "gn139 HCA-1"
> [1](237dffff94d829) "S-0019bbfffff8fac7"[16] # lid 89 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 10 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff94d813
> caguid=0x237dffff94d8l0
> Ca 2 "H - 00237dffff94d810" # "gn138 HCA-1"
> [1](237dffff94d811) "S-0019bbfffff8fac7"[15] # lid 84 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 10 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff94d817
> caguid= Ox237dffff94d8l4
> Ca 2 "H-00237dffff94d814" # "gn137 HCA-1"
> [1](237dffff94d815) "S-0019bbfffff8fac7"[14] # lid 85 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 10 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= Ox237dffff94c8ef
> caguid=Ox237dffff94c8ec
> Ca 2 "H-00237dffff94c8ec" # ^gn136 HCA-1"
> [1](237dffff94c8ed) "S-0019bbfffff8fac7"[13] # lid 77 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 10 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= 0x237dffff94d84b
> caguid= 0x237dffff94d848
> Ca 2 "H-00237dffff94d848" # "gn135 HCA-1"
> [1](237dffff94d849) "S-0019bbfffff8fac7"[12] # lid 95 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 10 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff94a8e7
> caguid=0x237dffff94a8e4
> Ca 2 "H- 00237dffff94a8e4" # "gn134 HCA-1"
> [1](237dffff94a8e5) "S-0019bbfffff8fac7"[11] # lid 73 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 10 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= OxleObffff84be8b
> caguid=Oxleobffff84be88
> Ca 2 "H- 001eObffff84be88" # "gn133 HCA-1"
> [1](leObffff84be89) "S-0019bbfffff8fac7"[10] # lid 25 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 10 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= Oxleobffff84be8f
> caguid=Oxleobffff84be8c
> Ca 2 "H-OOleObffff84be8c" # "gn132 HCA-1"
> [1](leobffff84be8d) "S-0019bbfffff8fac7"[9] # lid 26 lmc 0
> "MT47396 Infiniscale - III Mellanox Technologies" lid 10 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid=OxleObffff84aed7
> caguid= Oxleobffff84aed4
> Ca 2 "H - 001eobffff84aed4" # "gn163 HCA-1"
> [1](leobffff84aed5) "S-0019bbfffff8eac7"[24] # lid 18 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 8 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= OxleObffff84be9f
> caguid=Oxleobffff84be9c
> Ca 2 "H-001eObffff84be9c" # "gn161 HCA-1"
> [1](leobffff84be9d) "S-0019bbfffff8eac7"[22] # lid 30 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 8 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= 0x237dffff94982b
> caguid= Ox237dffff949828
> Ca 2 "H - 00237dffff949828" # "gn16O HCA-1"
> [1](237dffff949829) "S-0019bbfffff8eac7"[21] # lid 69 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 8 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= 0x237dffff94c857
> caguid= 0x237dffff94c854
> Ca 2 "H - 00237dffff94c854" # "gn159 HCA-1"
> [1](237dffff94c855) "S-0019bbfffff8eac7"[20] # lid 76 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 8 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid=0x237dffff94d807
> caguid=0x237dffff94d804
> Ca 2 "H-00237dffff94d804" # "gn158 HCA-1"
> [1](237dffff94d805) "S-0019bbfffff8eac7"[19] # lid 82 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 8 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid=0x237dffff94981f
> caguid=0x237dffff94981c
> Ca 2 "H-00237dffff94981c" # "gn157 HCA-1"
> [1](237dffff94981d) "S-0019bbfffff8eac7"[18] # lid 68 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 8 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid=0x237dffff 949803
> caguid=Ox237dffff949800
> Ca 2 "H-00237dffff949800" # "gn156 HCA-1"
> [1](237dffff949801) "S-0019bbfffff8eac7"[17] # lid 66 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 8 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid=Oxleobffff84bec7
> caguid=Oxleobffff84bec4
> Ca 2 "H-001eobffff84bec4" # "gn155 HCA-1"
> [1](leobffff84bec5) "S-0019bbfffff8eac7"[16] # lid 34 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 8 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= Oxleobffff84aec7
> caguid=Oxleobffff84aec4
> Ca 2 "H - 001eobffff84aec4" # "gn154 HCA-1"
> [1](leobffff84aec5) "S-0019bbfffff8eac7"[15] # lid 17 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 8 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid=Oxleobffff84aeaf
> caguid=Oxleobffff84aeac
> Ca 2 "H-001eobffff84aeac" # "gn153 HCA-1"
> [11(leobffff84aead) "S-0019bbfffff8eac7"[14] # lid 16 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 8 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid=Ox237dffff94981b
> caguid=Ox237dffff949818
> Ca 2 "H-00237dffff949818" # "gn152 HCA-1"
> [1](237dffff949819) "S-0019bbfffff8 d7"[13] # lid 67 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 8 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid=Oxleobffff84aee3
> caguid=Oxleobffff84aee0
> Ca 2 "H-001eobffff84aeeO" # "gn151 HCA-1"
> [1](leobffff84aeel) "S-0019bbfffff8eac7"[12] # lid 19 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 8 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid=Oxleobffff84ce93
> caguid=Oxleobffff84ce90
> Ca 2 "H-001eObffff84ce9O" # "gn150 HCA-1"
> [1](leObffff84ce91) "S-0019bbfffff8eac7"[11] # lid 39 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 8 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid=0x237dffff94b8lf
> caguid=Ox237dffff94b8lc
> Ca 2 "H-00237dffff94b81c" # "gn149 HCA-1"
> [1](237dffff94b81d) "S-0019bbfffff8eac7"[10] # lid 75 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 8 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid=0x237dffff94a8ff
> caguid=0x237dffff94a8fc
> Ca 2 "H-00237dffff94a8fc" # "gn148 HCA-1"
> [1] (237dffff94a8fd) "S-0019bbfffff8eac7" [9] # lid 74 lmc 0
> "MT47396 Infiniscale-III Mellanox Technologies" lid 8 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid=Oxleobffff84eclf
> caguid=OxleObffff84eclc
> Ca 2 "H-001eObffff84ec1c" # "gn195 HCA-1"
> [1](leObffff84ecld) "S-00237dffff9427b3"[24] # lid 41 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 55 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid=Ox237dffff94991b
> caguid=Ox237dffff949918
> Ca 2 "H-00237dffff949918" # "gn194 HCA-1"
> [1](237dffff949919) "S-00237dffff9427b3"[23] # lid 71 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 55 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid=Oxleobffff84ccd3
> caguid=OxleObffff84ccd0
> Ca 2 "H-001eobffff84ccd0" # "gn193 HCA-1"
> [1](leobffff84ccdl) "S-00237dffff9427b3"[22] # lid 37 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 55 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid=0x237dffff944933
> caguid=0x237dffff944930
> Ca 2 "H-00237dffff944930" # "gn192 HCA-1"
> [1](237dffff944931) "S-00237dffff9427b3"[21] # lid 62 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 55 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= Oxleobffff84ec47
> caguid=OxleObffff84ec44
> Ca 2 "H - 001eobffff84ec44" # "gn191 HCA-1"
> [1](leobffff84ec45) "S-00237dffff9427b3"[20] # lid 44 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 55 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= 0x237dffff94493b
> caguid=0x237dffff944938
> Ca 2 "H-00237dffff944938" # "gn190 HCA-1"
> [1](237dffff944939) "S-00237dffff9427b3"[19] # lid 63 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 55 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= Ox237dffff949907
> caguid=0x237dffff949904
> Ca 2 "H - 00237dffff949904" # "gn189 HCA-1"
> [1)(237dffff949905) "S-00237dffff9427b3"[18] # lid 70 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 55 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= 0xle0bffff84ec4b
> caguid=Oxleobffff84ec48
> Ca 2 "H-001e0bffff84ec48" # "gn188 HCA-1"
> [1](leobffff84ec49) "S-00237dffff9427b3"[17] # lid 45 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 55 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff9429db
> caguid=0x237dffff9429d8
> Ca 2 "H- 00237dffff9429d8" # "gn187 HCA-1"
> [1](237dffff9429d9) "S-00237dffff9427b3"[16] # lid 56 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 55 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= OxleObffff84ec37
> caguid=Oxleobffff84ec34
> Ca 2 "H-001eObffff84ec34" # "gn186 HCA-1"
> [1](leobffff84ec35) "S-00237dffff9427b3"[15] # lid 42 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 55 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff941b93
> caguid=0x237dffff941b90
> Ca 2 "H - 00237dffff941b90" # "gn185 HCA-1"
> [1](237dffff941b91) "S-00237dffff9427b3"[14] # lid 49 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 55 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= Oxleobffff84ec3b
> caguid= Oxleobffff84ec38
> Ca 2 "H - 001eObffff84ec38" # "gn184 HCA-1"
> [1](leObffff84ec39) "S-00237dffff9427b3"[13] # lid 43 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 55 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= 0x237dffff94lb8f
> caguid= 0x237dffff94lb8c
> Ca 2 "H-00237dffff941b8c" # "gn183 HCA-1"
> [1](237dffff94lb8d) "S-00237dffff9427b3"[12] # lid 48 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 55 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff94lb8b
> caguid=0x237dffff941b88
> Ca 2 "H-00237dffff941b88" # "gn182 HCA-1"
> [1](237dffff941b89) "S-00237dffff9427b3"[11] # lid 47 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 55 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid=OxleObffff84ec67
> caguid= OxleObffff84ec64
> Ca 2 "H-001eObffff84ec64" # "gn181 HCA-1"
> [1](leobffff84ec65) "S-00237dffff9427b3"[10] # lid 46 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 55 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff941bab
> caguid=0x237dffff941ba8
> Ca 2 "H - 00237dffff941ba8" # "gn180 HCA-1"
> [1](237dffff941ba9) "S-00237dffff9427b3"[9] # lid 50 lmc 0
> "MT47396 Infiniscale - III Mellanox Technologies" lid 55 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff94d837
> caguid= 0x237dffff94d834
> Ca 2 "H - 00237dffff94d834" # "gn131 HCA-1"
> [1](237dffff94d835) "S-0019bbfffff8eade"[24] # lid 92 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 9 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff94d827
> caguid= 0x237dffff94d824
> Ca 2 "H-00237dffff94d824" # "gn130 HCA-1"
> [1](237dffff94d825) "S-0019bbfffff8eade"[23] # lid 88 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 9 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff94d8lb
> caguid=0x237dffff94d818
> Ca 2 "H-00237dffff94d818" # ' pi`29 HCA-1"
> [1](237dffff94d819) "S-0019bbfffff8eade"[22]. # lid 86 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 9 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= OxleObffff84be6f
> caguid=Oxleobffff84be6c
> Ca 2 "H- 001eobffff84be6c" # "gn128 HCA-1"
> [1](leobffff84be6d) "S-0019bbfffff8eade"[21] # lid 20 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 9 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= OxleObffff84be7b
> caguid= OxleObffff84be78
> Ca 2 "H-001eobffff84be78" # "gn127 HCA-1"
> [11(leObffff84be79) "S-0019bbfffff8eade"[20] # lid 109
> lmc 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 9 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= OxleObffff84bea3
> caguid=Oxleobffff84bea0
> Ca 2 "H- 001eObffff84bea0" # "gn126 HCA-1"
> [1](leobffff84beal) "S-0019bbfffff8eade"[19] # lid 31 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 9 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= OxleObffff84be97
> caguid= Oxleobffff84be94
> Ca 2 "H-001eobffff84be94" # "gn125 HCA-1"
> [1](leobffff84be95) "S-0019bbfffff8eade"[18] # lid 28 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 9 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= OxleObffff84be77
> caguid=Oxleobffff84be74
> Ca 2 "H - 001eobffff84be74" # "gnl24 HCA-1"
> [1](leObffff84be75) "S-0019bbfffff8eade"[17] # lid 21 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 9 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff94d82f
> caguid= Ox237dffff94d82c
> Ca 2 "H-00237dffff94d82c" # "gn123 HCA-1"
> [1](237dffff94d82d) "S-0019bbfffff8eade"[16] # lid 90 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 9 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= Ox237dffff94c8ff
> caguid=Ox237dffff94c8fc
> Ca 2 "H- 00237dffff94c8fc" # "gn122 HCA-1"
> [1](237dffff94c8fd) "S-0019bbfffff8eade"[15] # lid 81 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 9 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= Ox237dffff94d83f
> caguid= Ox237dffff94d83c
> Ca 2 "H-00237dffff94d83c" # "gn121 HCA-1"
> [1](237dffff94d83d) "S-0019bbfffff8eade"[14] # lid 93 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 9 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff94d833
> caguid=0x237dffff94d830
> Ca 2 "H-00237dffff94d830" # "gn120 HCA-1"
> [1](237dffff94d831) "S-0019bbfffff8eade"[13] # lid 91 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 9 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff94d823
> caguid=0x237dffff94d820
> Ca 2 "H- 00237dffff94d820" # "gn119 HCA-1"
> [1](237dffff94d821) "S-0019bbfffff8eade"[12] # lid 87 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 9 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff94d8Of
> caguid= Ox237dffff94d8Oc
> Ca 2 "H- 00237dffff94d8Oc" # "gn118 HCA-1"
> [1)(237dffff94d8Od) "S-0019bbfffff8eade"[11] # lid 83 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 9 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= Oxleobffff84be83
> caguid=Oxleobffff84be80
> Ca 2 "H- 001eObffff84be8O" # "gn117 HCA-1"
> [11(leObffff84be81) "S-0019bbfffff8eade"[10] # lid 23 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 9 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= OxleObffff84beb7
> caguid= Oxleobffff84beb4
> Ca 2 "H-001eObffff84beb4" # "gn116 HCA-1"
> [1](leobffff84beb5) "S-0019bbfffff8eade"[9] # lid 32 lmc 0
> "MT47396 Infiniscale - III Mellanox Technologies" lid 9 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid=Oxleobffff84becb
> caguid=Oxleobffff84bec8
> Ca 2 "H-001eobffff84bec8" # "gn115 HCA-1"
> [11(leobffff84bec9) "S-0019bbfffff8eab7"[24] # lid 35 lmc
> 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 7 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= Ox237dffff94f8b7
> caguid=0x237dffff94f8b4
> Ca 2 "H - 00237dffff94f8b4" # "gn114 HCA-1"
> [1](237dffff94f8b5) "S-0019bbfffff8eab7"[23] # lid 98 lmc
> 0 "MT47396 Infiniscale - III Mellanox Tecologies" lid 7 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid=Oxleobffff84aea7
> caguid=Oxleobffff84aea4
> Ca 2 "H-001eobffff84aea4" # "gn113 HCA-1"
> [1](leobffff84aea5) "S-0019bbfffff8eab7"[22] # lid 110
> lmc 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 7 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= OxleObffff849d9f
> caguid=OxleObffff849d9c
> Ca 2 "H- 001eObffff849d9c" # "gn112 HCA-1"
> [1](leObffff849d9d) "S-0019bbfffff8eab7"[21] # lid 15 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 7 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= 0x237dffff94f8fb
> caguid=Ox237dffff94f8f8
> Ca 2 "H-00237dffff94f8f8" # "gnlll HCA-1"
> [1] (237dffff94f8f9) "S-0019bbfffff8eab7" [20] # lid 103
> lmc 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 7 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid=Ox237dffff94e82f
> caguid=Ox237dffff94e82c
> Ca 2 "H-00237dffff94e82c" # "gn110 HCA-1"
> [1](237dffff94e82d) "S-0019bbfffff8eab7"[19] # lid 97 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 7 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= Oxleobffff84bebf
> caguid= Oxleobffff84bebc
> Ca 2 "H-001eObffff84bebc" # "gn109 HCA-1"
> [1](leObffff84bebd) "S-0019bbfffff8eab7"[18] # lid 33 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 7 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= 0x237dffff94f8e3
> caguid=Ox237dffff94f8e0
> Ca 2 "H - 00237dffff94f8eO" # "gn108 HCA-1"
> [1](237dffff94f8el) "S-0019bbfffff8eab7"[17] # lid 99 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 7 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= Ox237dffff94f8ef
> caguid=Ox237dffff94f8ec
> Ca 2 "H - 00237dffff94f8ec" # "gn107 HCA-1"
> [1](237dffff94f8ed) "S-0019bbfffff8eab7"[16] # lid 100
> lmc 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 7 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= Ox237dffff94f8f3
> caguid=Ox237dffff94f8fO
> Ca 2 "H-00237dffff94f8fO" # "gn106 HCA-1"
> [1](237dffff94f8fl) "S-0019bbfffff8eab7"[15] # lid 101
> lmc 0 "MT47396 Infiniscale-III Mellanox Technologies" lid 7 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid=Oxleobffff842c87
> caguid= OxleObffff842c84
> Ca 2 "H-001eObffff842c84" # "gn105 HCA-1"
> [1](leObffff842c85) "S-0019bbfffff8eab7"[14] # lid 13 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 7 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= OxleObffff84cel7
> caguid=OxleObffff84cel4
> Ca 2 "H-001eObffff84ce14" # "gn104 HCA-1"
> [1](leObffff84cel5) "S-0019bbfffff8eab7"[13] # lid 38 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 7 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= Ox237dffff94f8f7
> caguid=Ox237dffff94f8f4
> Ca 2 "H - 00237dffff94f8f4" # "gn103 HCA-1"
> [1](237dffff94f8f5) "S-0019bbfffff8eab7"[12] # lid 102
> lmc 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 7 4xDDR
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= OxleObffff849d93
> caguid=OxleObffff849d90
> Ca 2 "H- 001eObffff849d9O" # "gn102 HCA-1" [1](leobffff849d91)
> "S-0019bbfffff8eab7"[11] # lid 14 lmc 0 "MT47396 Infiniscale - III Mellanox
> Technologies" lid 7 4xDDR
>
>
> vendid=0x2c9
> devid=0x634a
> sysimgguid= Oxle0bffff84bed7
> caguid=Oxleobffff84bed4
> Ca 2 "H-001eObffff84bed4" # "gn101 HCA-1"
> [1](leobffff84bed5) "S-0019bbfffff8eab7"[10] # lid 36 lmc
> 0 "MT47396 Infiniscale - III Mellanox Technologies" lid 7 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= Ox237dffff94e8lf
> caguid=Ox237dffff94e8lc
> Ca 2 "H-00237dffff94e81c" # "gn100 HCA-1"
> [1](237dffff94e8ld) "S-0019bbfffff8eab7"[9] # lid 96 lmc 0
> "MT47396 Infiniscale-III Mellanox Technologies" lid 7 4xDDR
>
> vendid=Ox3ba
> devid= 0x634a
> sysimgguid= Ox3ba00010OdOa7
> caguid=Ox3ba000100d0a4
> Ca 2 "H - 0003ba000100dOa4" # "Sun Microsystems InfiniBand Eagle
> DDR HCA"
> [1](3ba000100dOa5) "S-0008fl040041811a"[14] # lid 2 lmc
> 0 "ISR9024D Voltaire" lid 6 4xDDR
> [2](3ba000100dOa6) "S-0008f1040041a"[13] # lid 3 lmc
> 0 "ISR9024D Voltaire" lid 6 4xDDR
>
> vendid=0x2c9
> devid= 0x634a
> sysimgguid= Oxleobffff4c0827
> caguid= OxleObffff4cO824
> Ca 2 "H-001e0bffff4cO824" # "headnode HCA-1"
> [1](leObffff4cO825) "S-0008f1040041811a"[12] # lid 12 lmc
> 0 "ISR9024D Voltaire" lid 6 4xDDR
>
>