[ofa-general] Bandwidth of performance with multirail IB

[Davis, Arlin R]

Jie,

>>> The peak bandwidth I can get is ~ 1.3 GB/s (not 
>bi-directional), which
>>> is almost the same as single rail connections.
>>>     

That is correct 10Gb/s each port for a max of 20Gb/s.

>>
>> Assuming you have a 2.5 GT/s pci-express x8 that speed is a 
>result of the bus 
>> not being able to keep up with the HCA. Since the bus is 
>holding even a 
>> single DDR IB port back you see no improvement with two ports.
>>
>>   
>I do connect HCA in a 16x pci-e slot on each node.
>However, I am trying to drive 2 ports simultaneously.
>
>The workstation i am using is Sun Ultra 24,
>and the HCA is Mellanox ConnectX  MHGH28-XTC.
>The data for the HCA and Ultra 24 is
>
>MHGH28-XTC 
>IB ports: Dual Copper 4X 20Gb/s 
>Host Bus: PCIe 2.0 2.5GT/s

Peter is right. Your maximum speed given this configuration
is 20Gb/s (40Gb/s bi-directional) since you have an 8x adapter 
in a 2.5GT/s bus. 

It doesn't matter that you have it in a 16x slot, your
adapter is only 8x 2.5GT/s.

-arlin