<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
john t wrote:
<blockquote
cite="mida94efc20610060209q3698cb83xc63ec39654494ad3@mail.gmail.com"
type="cite">
<div>Hi Shannon,</div>
<div> </div>
<div>The bandwidth figures that you quoted below match with my
readings for single port Mellanox DDR HCA (both for unidirection and
bidirection). So it seems dual port SDR HCA performs as good as single
port DDR HCA. It would help if you can also tell the bandwidth that you
got using one port of your dual-port SDR HCA card. Was it half the
bandwidth that you stated below, which means having two SDR ports per
HCA helps.</div>
</blockquote>
<br>
For a single PCIe Mellanox SDR HCA port, most architectures we've
tested provide 960-970 MB/sec unidirectional and 1800-1900 MB/sec
bidirectional bandwidth using MPI.<br>
<br>
Shannon<br>
<br>
<br>
<blockquote
cite="mida94efc20610060209q3698cb83xc63ec39654494ad3@mail.gmail.com"
type="cite">
<div></div>
<div> In my case it seems having two ports (DDR) per HCA does not
increase BW, since PCI-e x8 limit is 16 Gb/sec per direction and each
of the two HCA ports (DDR) though capable of transferring 16 Gb/sec in
each direction, when used together can not go above 16 Gb/sec.
</div>
<div> </div>
<div>Regards,</div>
<div>John T.<br>
<br>
</div>
<div><span class="gmail_quote">On 10/5/06, <b
class="gmail_sendername">Shannon V. Davidson</b> <<a
href="mailto:svdavidson@charter.net">svdavidson@charter.net</a>>
wrote:</span>
<blockquote class="gmail_quote"
style="border-left: 1px solid rgb(204, 204, 204); margin: 0px 0px 0px 0.8ex; padding-left: 1ex;">
<div text="#000000" bgcolor="#ffffff">John,<br>
<br>
In our testing with dual port Mellanox SDR HCAs, we found that not all
PCI-express implementations are equal. Depending on the PCIe chipset,
we measured unidirectional SDR dual-rail bandwidth ranging from
1100-1500 MB/sec and bidirectional SDR dual-rail bandwidth ranging from
1570-2600 MB/sec. YMMV, but had good luck with Intel and Nvidia
chipsets, and less success with the Broadcom Serverworks HT-1000 and
HT-2000 chipsets. My last report (in June 2006) was that Broadcom was
working to improve their PCI-express performance.
<br>
<br>
Regards,<br>
Shannon<br>
<br>
john t wrote:
<blockquote
cite="http://mida94efc20610050518s40ca3a80xe6c56034df157be4@mail.gmail.com"
type="cite">
<div><span class="e" id="q_10e18f4859f18a53_1">
<div>Hi Bernard,</div>
<div> </div>
<div>I had a configuration issue. I fixed it and now I get same
BW (i.e. around 10 Gb/sec) on each port provided I use ports on
different HCA cards. If I use two ports of the same HCA card then BW
gets divided between these two ports. I am using Mellanox HCA cards and
doing simple send/recv using uverbs. </div>
<div> </div>
<div>Do you think it could be an issue with Mallanox driver or
could it be due to system/PCI-E limitation.</div>
<div> </div>
<div>Regards,</div>
<div>John T.<br>
<br>
</div>
<div><span class="gmail_quote">On 10/3/06, <b
class="gmail_sendername">Bernard King-Smith</b> <<a
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:wombat2@us.ibm.com" target="_blank">wombat2@us.ibm.com </a>>
wrote:</span>
<blockquote class="gmail_quote"
style="border-left: 1px solid rgb(204, 204, 204); margin: 0px 0px 0px 0.8ex; padding-left: 1ex;"><br>
<font face="sans-serif" size="2"><span name="st">John</span>,</font>
<br>
<br>
<font face="sans-serif" size="2">Who's adapter (manufacturer)
are you using? It is usually an adapter implementation or driver issue
that occures when you cannot scale across multiple links. The fact that
you don't scale up from one link, but it appears they share a fixed
bandwidth across N links means that there is a driver or stack issue.
At one time I think that IPoIB and maybe other IB drivers used only one
event queue across multiple links which would be a bottleneck. We added
code in the IBM EHCA driver to get round this bottleneck. </font><br>
<br>
<font face="sans-serif" size="2">Are your measurements using
MPI or IP. Are you using separate tasks/sockets per link and using
different subnets if using IP?</font> <br>
<font face="sans-serif" size="2"><br>
Bernie King-Smith <br>
IBM Corporation<br>
Server Group<br>
Cluster System Performance <br>
<a onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:wombat2@us.ibm.com" target="_blank">wombat2@us.ibm.com</a>
(845)433-8483 <br>
Tie. 293-8483 or wombat2 on NOTES <br>
<br>
"We are not responsible for the world we are born into, only for the
world we leave when we die.<br>
So we have to accept what has gone before us and work to change the
only thing we can, <br>
-- The Future." William Shatner</font> <br>
<br>
<tt><font size="2"><span name="st">john</span> t" <<a
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:johnt1johnt2@gmail.com" target="_blank">
johnt1johnt2@gmail.com</a>> wrote on 10/03/2006 09:42:24 AM:</font></tt>
<div><span><br>
<tt><font size="2">> <br>
> Hi,</font></tt> <br>
<tt><font size="2">> </font></tt> <br>
<tt><font size="2">> I have two HCA cards, each having two
ports and each connected to a <br>
> separate PCI-E x8 slot. </font></tt><br>
<tt><font size="2">> </font></tt> <br>
<tt><font size="2">> Using one HCA port I get end to end BW
of 11.6 Gb/sec (uni-direction RDMA).</font></tt> <br>
<tt><font size="2">> If I use two ports of the same HCA or
different HCA, I get between 5 <br>
> to 6.5 Gb/sec point-to-point BW on each port. BW on each port <br>
> further reduces if I use more ports. I am not able to understand <br>
> this behaviour. Is there any limitation on max. BW that a system
can <br>
> provide? Does the available BW get divided among multiple HCA
ports <br>
> (which means having multiple ports will not increase the BW)? </font></tt><br>
<tt><font size="2">> </font></tt> <br>
<tt><font size="2">> </font></tt> <br>
<tt><font size="2">> Regards,</font></tt> <br>
<tt><font size="2">> John T<br>
</font></tt></span></div>
</blockquote>
</div>
<br>
</span></div>
<pre><hr size="4" width="90%">
_______________________________________________
openib-general mailing list
<a onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:openib-general@openib.org" target="_blank">openib-general@openib.org</a>
<a onclick="return top.js.OpenExtLink(window,event,this)"
href="http://openib.org/mailman/listinfo/openib-general"
target="_blank">http://openib.org/mailman/listinfo/openib-general</a>
To unsubscribe, please visit <a
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://openib.org/mailman/listinfo/openib-general"
target="_blank">http://openib.org/mailman/listinfo/openib-general</a></pre>
</blockquote>
<br>
<br>
<pre cols="72">--
____________________________________________
Shannon V. Davidson <a
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:svdavidson@charter.net" target="_blank"><svdavidson@charter.net></a>
Senior Software Engineer Raytheon
636-479-7465 office 443-383-0331 fax
____________________________________________
</pre>
</div>
</blockquote>
</div>
<br>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
____________________________________________
Shannon V. Davidson <a class="moz-txt-link-rfc2396E" href="mailto:svdavidson@charter.net"><svdavidson@charter.net></a>
Senior Software Engineer Raytheon
636-479-7465 office 443-383-0331 fax
____________________________________________
</pre>
</body>
</html>