[openib-general] Multi-port HCA
Shannon V. Davidson
svdavidson at charter.net
Fri Oct 6 12:40:09 PDT 2006
john t wrote:
> Hi Shannon,
>
> The bandwidth figures that you quoted below match with my readings for
> single port Mellanox DDR HCA (both for unidirection and bidirection).
> So it seems dual port SDR HCA performs as good as single port DDR HCA.
> It would help if you can also tell the bandwidth that you got using
> one port of your dual-port SDR HCA card. Was it half the bandwidth
> that you stated below, which means having two SDR ports per HCA helps.
For a single PCIe Mellanox SDR HCA port, most architectures we've tested
provide 960-970 MB/sec unidirectional and 1800-1900 MB/sec bidirectional
bandwidth using MPI.
Shannon
> In my case it seems having two ports (DDR) per HCA does not increase
> BW, since PCI-e x8 limit is 16 Gb/sec per direction and each of the
> two HCA ports (DDR) though capable of transferring 16 Gb/sec in each
> direction, when used together can not go above 16 Gb/sec.
>
> Regards,
> John T.
>
>
> On 10/5/06, *Shannon V. Davidson* <svdavidson at charter.net
> <mailto:svdavidson at charter.net>> wrote:
>
> John,
>
> In our testing with dual port Mellanox SDR HCAs, we found that not
> all PCI-express implementations are equal. Depending on the PCIe
> chipset, we measured unidirectional SDR dual-rail bandwidth
> ranging from 1100-1500 MB/sec and bidirectional SDR dual-rail
> bandwidth ranging from 1570-2600 MB/sec. YMMV, but had good luck
> with Intel and Nvidia chipsets, and less success with the Broadcom
> Serverworks HT-1000 and HT-2000 chipsets. My last report (in June
> 2006) was that Broadcom was working to improve their PCI-express
> performance.
>
> Regards,
> Shannon
>
> john t wrote:
>> Hi Bernard,
>>
>> I had a configuration issue. I fixed it and now I get same BW
>> (i.e. around 10 Gb/sec) on each port provided I use ports on
>> different HCA cards. If I use two ports of the same HCA card then
>> BW gets divided between these two ports. I am using Mellanox HCA
>> cards and doing simple send/recv using uverbs.
>>
>> Do you think it could be an issue with Mallanox driver or could
>> it be due to system/PCI-E limitation.
>>
>> Regards,
>> John T.
>>
>>
>> On 10/3/06, *Bernard King-Smith* <wombat2 at us.ibm.com
>> <mailto:wombat2 at us.ibm.com>> wrote:
>>
>>
>> John,
>>
>> Who's adapter (manufacturer) are you using? It is usually an
>> adapter implementation or driver issue that occures when you
>> cannot scale across multiple links. The fact that you don't
>> scale up from one link, but it appears they share a fixed
>> bandwidth across N links means that there is a driver or
>> stack issue. At one time I think that IPoIB and maybe other
>> IB drivers used only one event queue across multiple links
>> which would be a bottleneck. We added code in the IBM EHCA
>> driver to get round this bottleneck.
>>
>> Are your measurements using MPI or IP. Are you using separate
>> tasks/sockets per link and using different subnets if using IP?
>>
>> Bernie King-Smith
>> IBM Corporation
>> Server Group
>> Cluster System Performance
>> wombat2 at us.ibm.com <mailto:wombat2 at us.ibm.com> (845)433-8483
>> Tie. 293-8483 or wombat2 on NOTES
>>
>> "We are not responsible for the world we are born into, only
>> for the world we leave when we die.
>> So we have to accept what has gone before us and work to
>> change the only thing we can,
>> -- The Future." William Shatner
>>
>> john t" < johnt1johnt2 at gmail.com
>> <mailto:johnt1johnt2 at gmail.com>> wrote on 10/03/2006 09:42:24
>> AM:
>>
>> >
>> > Hi,
>> >
>> > I have two HCA cards, each having two ports and each
>> connected to a
>> > separate PCI-E x8 slot.
>> >
>> > Using one HCA port I get end to end BW of 11.6 Gb/sec
>> (uni-direction RDMA).
>> > If I use two ports of the same HCA or different HCA, I get
>> between 5
>> > to 6.5 Gb/sec point-to-point BW on each port. BW on each port
>> > further reduces if I use more ports. I am not able to
>> understand
>> > this behaviour. Is there any limitation on max. BW that a
>> system can
>> > provide? Does the available BW get divided among multiple
>> HCA ports
>> > (which means having multiple ports will not increase the BW)?
>> >
>> >
>> > Regards,
>> > John T
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> openib-general mailing list
>> openib-general at openib.org <mailto:openib-general at openib.org>
>> http://openib.org/mailman/listinfo/openib-general
>>
>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>
>
> --
> ____________________________________________
>
> Shannon V. Davidson <svdavidson at charter.net> <mailto:svdavidson at charter.net>
> Senior Software Engineer Raytheon
> 636-479-7465 office 443-383-0331 fax
> ____________________________________________
>
>
>
>
--
____________________________________________
Shannon V. Davidson <svdavidson at charter.net>
Senior Software Engineer Raytheon
636-479-7465 office 443-383-0331 fax
____________________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20061006/59a626f5/attachment.html>
More information about the general
mailing list