<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">Hi </span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">If the point 2 is
true, we DO did the test with the WinSocket based code (we use .Net Framework
4.0 and use IO Completion port to send/receive the data in async mode), what
the problem mentioned is that:</span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-align:justify;
text-indent:-18.0pt;mso-list:l0 level1 lfo1"><span lang="EN-US" style="mso-fareast-font-family:"Times New Roman";color:#1F497D"><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span></span><span lang="EN-US" style="color:#1F497D">I run my
testing code try to send large data set and the performance throught can’t
reach 300MB/s;</span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-align:justify;
text-indent:-18.0pt;mso-list:l0 level1 lfo1"><span lang="EN-US" style="mso-fareast-font-family:"Times New Roman";color:#1F497D"><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span></span><span lang="EN-US" style="color:#1F497D">I run the
SDPConnect.exe tool</span><span style="font-family:宋体;color:#1F497D">（</span><span lang="EN-US" style="color:#1F497D">It can be found in server installed in
MLNX_VPI/IB/Tools folder</span><span style="font-family:宋体;color:#1F497D">）</span><span lang="EN-US" style="color:#1F497D"> use Overlapped send and receive in both side
of the testing machine, it reached 900+MB/s throughput performance result.</span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D"> </span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1F497D">So it turns out that
we must seek more information from documentation or SDK for how to write such
WinSocket based App, otherwise we just use InfiniBand as a normal IP based LAN.</span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";
color:#1F497D"> </span></p><br><div class="gmail_quote">On Tue, Jan 25, 2011 at 5:26 PM, Tzachi Dar <span dir="ltr"><<a href="mailto:tzachid@mellanox.co.il">tzachid@mellanox.co.il</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Hi,</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">It seems from your question that there is some confusion:</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">There are two ways to work with IB:</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p><span style="font-size:11.0pt;color:#1F497D"><span>1)<span style="font:7.0pt "Times New Roman"">
</span></span></span><span dir="LTR"></span><span style="font-size:11.0pt;color:#1F497D">Use native ib program.
This means that you have to write your code differently, in order for it to
work. To see examples of such programs you can download our code from svn://<a href="http://openib.tc.cornell.edu/gen1/trunk" target="_blank">openib.tc.cornell.edu/gen1/trunk</a>.
there are many examples there (trunk\tools\perftests\user\send_bw, or trunk\tests\perftest\send_bw).
In general in windows there are many apis that one can use. I’m currently not explaining
this thoroughly, because I believe that you want to go to option #2.</span></p>
<p><span style="font-size:11.0pt;color:#1F497D"><span>2)<span style="font:7.0pt "Times New Roman"">
</span></span></span><span dir="LTR"></span><span style="font-size:11.0pt;color:#1F497D">Use one of WSD or SDP.
With this approach you don’t have to change anything in your source code. In fact,
you don’t actually need to have the source code at all. For example the sdpconnect
program (source attached) is a native windows socket program. You can find more
information about how to run your program with one of this in the Mellanox documentation.
Please also read the limitations section, as both have their limitations. </span></p>
<p class="MsoNormal" style="margin-left:.25in"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">By the way, when you write “SDPConnect.exe showed great performance”,
how did you run it (wsd? SDP?)?</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Thanks</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Tzachi</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt">From:</span></b><span style="font-size:10.0pt">
<a href="mailto:ofw-bounces@lists.openfabrics.org" target="_blank">ofw-bounces@lists.openfabrics.org</a> [mailto:<a href="mailto:ofw-bounces@lists.openfabrics.org" target="_blank">ofw-bounces@lists.openfabrics.org</a>] <b>On
Behalf Of </b>suo zhang<br>
<b>Sent:</b> Tuesday, January 25, 2011 10:13 AM<br>
<b>To:</b> <a href="mailto:ofw@lists.openfabrics.org" target="_blank">ofw@lists.openfabrics.org</a><br>
<b>Subject:</b> Re: [ofw] WSD Detail Sample Code</span></p>
</div>
</div><div><div></div><div class="h5">
<p class="MsoNormal"> </p>
<p class="MsoNormal"><span><span style="font-size:10.0pt">Hi</span></span><span style="font-size:10.0pt"><br>
<br>
<span>Following is our question.</span><br>
<br>
<span>Thanks</span><br>
<br>
<span>Suo</span><br>
<br>
<span>----------------------------------------------------------------------------</span><br>
<span>--------------------------------------------</span><br>
<br>
<br>
<br>
<span>We already wrote a IOCP and WinSocket based
communication program. The</span><br>
<span>testing upon Infiniband indicated that this
program still go over the</span><br>
<span>IPOverIP and has the overhead of Windows IP stack.</span><br>
<br>
<br>
<br>
<span>We saw this PPT and want to know how our WinSocket
based program can fully</span><br>
<span>utilize the capabilities of Inifniband RDMA and
how to write code over WSD</span><br>
<span>to avoid the overhead of Windows IP stack.</span><br>
<br>
<br>
<br>
<span>We just need a sample code like the source code of
tool</span><br>
<span>SDPConnect.exe(within the Mallanox driver tools
folder) etc. It showed great</span><br>
<span>performance we can’t achieved by our WinSocket
code.</span><br>
<br>
<br>
<br>
<span>We’ve got some progress improvement:</span><br>
<br>
<span>1. By searching the source
code in OpenFabric, it turns out that these</span><br>
<span>open source include some tools’ source code, like
this c file attached;</span><br>
<br>
<span>2. There was some code
founded in such tool’s source code like:</span><br>
<br>
<span>while (ccnt < user_param->iters || rcnt <
user_param->iters ) {</span><br>
<br>
<span> while (scnt
< user_param->iters && (scnt - ccnt) <</span><br>
<span>user_param->tx_depth / 2) {</span><br>
<br>
<span>
ib_send_wr_t *bad_wr;</span><br>
<br>
<span>
tposted[scnt] = get_cycles();</span><br>
<br>
<span>
ib_status = ib_post_send(qp, &ctx->wr, &bad_wr);</span><br>
<br>
<span>
if (ib_status != IB_SUCCESS) {</span><br>
<br>
<span>
fprintf(stderr, "Couldn't
post send: scnt=%d</span><br>
<span>ib_status %d\n",</span><br>
<br>
<span>
scnt,ib_status);</span><br>
<br>
<span>
return 1;</span><br>
<br>
<span>
}</span><br>
<br>
<span>
++scnt;</span><br>
<br>
<span>
PERF_DEBUG("scnt = %d \n",scnt);</span><br>
<br>
<span> }</span><br>
<br>
<br>
<br>
<span>But no document just have to guess from the source
code for what “qp”,</span><br>
<span>“ctx” means and how to use them. These “ib_”
initiated APIs were</span><br>
<span>absolutely NOT WinSocket APIs. What we asked for
is SDK or documents for how</span><br>
<span>to using such API and which Header files we should
include and which lib</span><br>
<span>have to be linked. A sample project will be OK if
no such document exists.</span><br>
<br>
<br>
<br>
<br>
</span>Thanks </p>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">Suo</p>
<div>
<p class="MsoNormal">2011/1/25 Smith, Stan <<a href="mailto:stan.smith@intel.com" target="_blank">stan.smith@intel.com</a>></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Please submit modified question to <a href="mailto:ofw@lists.openfabrics.org" target="_blank">ofw@lists.openfabrics.org</a></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Furthermore, please identify which
sample code you are referring to in order that someone can respond to your
question.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Which Windows OS and processor
(x86,x64,ia64).</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Generally speaking WSD (WinSock Direct)
is not used directly but utilized by opening a socket(AF_INET) which beneath
the socket layer that has a WSD provider enabled for the specific address
family.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">For winOFED 2.3 release, run ‘installsp
–l’ to see WSD providers.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">WSD in no longer enabled by default for
svr2008* – see ‘installsp –I’</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Stan.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">From: <b>Suo
Zhang</b> <<a href="mailto:zhangs@bjcarnation.com.cn" target="_blank">zhangs@bjcarnation.com.cn</a>><br>
Date: 2011/1/20<br>
Subject: WSD Detail Sample Code<br>
To: <a href="mailto:membership@openfabrics.org" target="_blank">membership@openfabrics.org</a></p>
<div>
<div>
<p class="MsoNormal">Hi
</p>
<p class="MsoNormal">When
I use WSD in my project, I found the sample code is very difficult to use.
Because the sample code is very simple.</p>
<p class="MsoNormal">I
want to know where can I get a detail sample code for WSD. Because my
application must use zero copy to get good performance.</p>
<p class="MsoNormal">Thanks</p>
<p class="MsoNormal">Best
regards</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"><span lang="ZH-CN" style="font-family:MingLiU">张锁</span> Suo Zhang</p>
<p class="MsoNormal">Sales
Engineer</p>
<p class="MsoNormal"><span lang="ZH-CN" style="font-family:"MS Gothic"">北京石竹科技有限公司</span><span lang="ZH-CN"> </span></p>
<p class="MsoNormal">Beijing
Carnation Technologies Limited</p>
<p class="MsoNormal">Mobile
+86-186-1175-6167</p>
<p class="MsoNormal">Tel<span lang="ZH-CN" style="font-family:"MS Gothic"">：</span>+8610-68587971-8029</p>
<p class="MsoNormal">Fax<span lang="ZH-CN" style="font-family:"MS Gothic"">：</span>+8610-68587975</p>
<p class="MsoNormal"><a href="http://www.vme.cn/" target="_blank">Http://www.vme.cn</a></p>
<p class="MsoNormal"> </p>
</div>
</div>
</div>
<p class="MsoNormal"> </p>
</div>
</div>
</div>
<p class="MsoNormal"> </p>
</div>
</div></div></div>
</div>
</div>
</blockquote></div><br>