[ewg] strange value of IB port rate

[Hal Rosenstock]

On 2/20/2013 7:41 AM, Mahesh Chaudhari wrote:
> We have a cluster of 16 nodes. And rest all 15 nodes shows the same
> value i.e. 10 Gb/sec (4X) except this node

What's different about this node ? Same HCAs ? Same firmware ? Same
kernel ? Same user space packages ?

IB spec says:
C14-24.2.1: If PortInfo:Portstate=Down, then
• a SubnGet(PortInfo) shall produce valid data for PortInfo:PortState
and PortInfo:PortPhysicalState; whether any other component has
valid data is vendor-dependent

It relies on those vendor dependent values. Some additional checking
could be introduced into those utilities to avoid this confusion but
they're just largely pretty printers for the returned information.

-- Hal

> 
> ------------------------------------------------------------------------
> *From:* Hal Rosenstock <hal at dev.mellanox.co.il>
> *To:* Mahesh Chaudhari <mahesh.chaudhari at ymail.com>
> *Cc:* "ewg at lists.openfabrics.org" <ewg at lists.openfabrics.org>
> *Sent:* Wednesday, 20 February 2013 5:24 PM
> *Subject:* Re: [ewg] strange value of IB port rate
> 
> On 2/20/2013 6:51 AM, Mahesh Chaudhari wrote:
>> Hi All,
>> i have Mellanox dual port IB card installed in m/c
>> when i do :
>>              "cat /sys/class/infiniband/mthca0/ports/1/rate "
>> it shows 20 Gb/sec (4X DDR)
>>
>> while on port2
>>                " cat /sys/class/infiniband/mthca0/ports/2/rate "
>> it shows 8.5 Gb/sec (4X)
>>
>> when i looked at source code, i found a equation to calculate rate :
>>
>>                        rate = 25 *
>> ib_width_enum_to_int(attr.active_width) * attr.active_speed;
>>                     
>>                        return sprintf(buf, "%d%s Gb/sec (%dX%s)\n",
>>                        rate / 10, rate % 10 ? ".5" : "",
>>                        ib_width_enum_to_int(attr.active_width), speed);
>>
>>
>>                                where 
>> ib_width_enum_to_int(attr.active_width) =  1 | 4 |8 |12 | -1(error)
>>                                              attr.active_speed = 1 | 2
> | 4
>>
>>
>>    i am wondering, how could it possible to get such an odd value (8.5
>> Gbps)
>>
>> /usr/ofed/sbin/ibstatus utility also shows :
>>
>> Infiniband device 'mthca0' port 1 status:
>>    default gid:    fe80:0000:0000:0000:001a:4bff:ff0c:96e5
>>    base lid:    0x6
>>    sm lid:        0x1
>>    state:        4: ACTIVE
>>    phys state:    5: LinkUp
>>    rate:        20 Gb/sec (4X DDR)
>>    link_layer:    InfiniBand
>>
>> Infiniband device 'mthca0' port 2 status:
>>    default gid:    fe80:0000:0000:0000:001a:4bff:ff0c:96e6
>>    base lid:    0x0
>>    sm lid:        0x0
>>    state:        1: DOWN
>>    phys state:    2: Polling
>>    rate:        8.5 Gb/sec (4X)
>>    link_layer:    InfiniBand
>>
>>
>>
>>
>> Any Clue ???
> 
> In the case of DOWN ports, rate is meaningless and should be ignored.
> 
> -- Hal
> 
>>
>>
>> _______________________________________________
>> ewg mailing list
>> ewg at lists.openfabrics.org <mailto:ewg at lists.openfabrics.org>
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
> 
> 
>