[ofw] WSD Detail Sample Code
suo zhang
zhangs at bjcarnation.com.cn
Thu Jan 27 05:18:52 PST 2011
Hi Tzachi
Following is the question with our customer. Mr. Hu is our customer who is
using WSD.
We’ve checked the .net IO Completion implementation, it still go through the
WinSocket and use Overlapped to do the work. What we mentioned is that the
performance is poor and it turns out using WinSocket still go through IP
stack. So we are trying to find what is the proper way to using ZCopy with
WSD/SDP.
Thanks
On Thu, Jan 27, 2011 at 12:40 AM, Tzachi Dar <tzachid at mellanox.co.il> wrote:
> Hi,
>
>
>
> I don’t think that we have ever tried to run neither wsd nor sdp on .net
> framework code, so I don’t know if it will work or not.
>
>
>
> In any case, wsd has performance counters that you can use to verify if the
> traffic is going through or not (under ib winsock direct).
>
> So please open this performance counters, and see if your program is using
> it or not?
>
>
>
> Thanks
>
> Tzachi
>
>
>
> *From:* lock.zhang at gmail.com [mailto:lock.zhang at gmail.com] *On Behalf Of *suo
> zhang
> *Sent:* Wednesday, January 26, 2011 8:25 AM
> *To:* Tzachi Dar
> *Cc:* ofw at lists.openfabrics.org
>
> *Subject:* Re: [ofw] WSD Detail Sample Code
>
>
>
> Hi
>
>
>
> If the point 2 is true, we DO did the test with the WinSocket based code
> (we use .Net Framework 4.0 and use IO Completion port to send/receive the
> data in async mode), what the problem mentioned is that:
>
> 1. I run my testing code try to send large data set and the
> performance throught can’t reach 300MB/s;
>
> 2. I run the SDPConnect.exe tool（It can be found in server installed
> in MLNX_VPI/IB/Tools folder） use Overlapped send and receive in both side
> of the testing machine, it reached 900+MB/s throughput performance result.
>
>
>
> So it turns out that we must seek more information from documentation or
> SDK for how to write such WinSocket based App, otherwise we just use
> InfiniBand as a normal IP based LAN.
>
>
>
>
>
> On Tue, Jan 25, 2011 at 5:26 PM, Tzachi Dar <tzachid at mellanox.co.il>
> wrote:
>
> Hi,
>
>
>
> It seems from your question that there is some confusion:
>
>
>
> There are two ways to work with IB:
>
>
>
> 1) Use native ib program. This means that you have to write your code
> differently, in order for it to work. To see examples of such programs you
> can download our code from svn://openib.tc.cornell.edu/gen1/trunk. there
> are many examples there (trunk\tools\perftests\user\send_bw, or
> trunk\tests\perftest\send_bw). In general in windows there are many apis
> that one can use. I’m currently not explaining this thoroughly, because I
> believe that you want to go to option #2.
>
> 2) Use one of WSD or SDP. With this approach you don’t have to change
> anything in your source code. In fact, you don’t actually need to have the
> source code at all. For example the sdpconnect program (source attached) is
> a native windows socket program. You can find more information about how to
> run your program with one of this in the Mellanox documentation. Please also
> read the limitations section, as both have their limitations.
>
>
>
> By the way, when you write “SDPConnect.exe showed great performance”, how
> did you run it (wsd? SDP?)?
>
>
>
> Thanks
>
> Tzachi
>
>
>
> *From:* ofw-bounces at lists.openfabrics.org [mailto:
> ofw-bounces at lists.openfabrics.org] *On Behalf Of *suo zhang
> *Sent:* Tuesday, January 25, 2011 10:13 AM
> *To:* ofw at lists.openfabrics.org
> *Subject:* Re: [ofw] WSD Detail Sample Code
>
>
>
> Hi
>
> Following is our question.
>
> Thanks
>
> Suo
>
>
> ----------------------------------------------------------------------------
> --------------------------------------------
>
>
>
> We already wrote a IOCP and WinSocket based communication program. The
> testing upon Infiniband indicated that this program still go over the
> IPOverIP and has the overhead of Windows IP stack.
>
>
>
> We saw this PPT and want to know how our WinSocket based program can fully
> utilize the capabilities of Inifniband RDMA and how to write code over WSD
> to avoid the overhead of Windows IP stack.
>
>
>
> We just need a sample code like the source code of tool
> SDPConnect.exe(within the Mallanox driver tools folder) etc. It showed
> great
> performance we can’t achieved by our WinSocket code.
>
>
>
> We’ve got some progress improvement:
>
> 1. By searching the source code in OpenFabric, it turns out that
> these
> open source include some tools’ source code, like this c file attached;
>
> 2. There was some code founded in such tool’s source code like:
>
> while (ccnt < user_param->iters || rcnt < user_param->iters ) {
>
> while (scnt < user_param->iters && (scnt - ccnt) <
> user_param->tx_depth / 2) {
>
> ib_send_wr_t *bad_wr;
>
> tposted[scnt] = get_cycles();
>
> ib_status = ib_post_send(qp, &ctx->wr, &bad_wr);
>
> if (ib_status != IB_SUCCESS) {
>
> fprintf(stderr, "Couldn't post send: scnt=%d
> ib_status %d\n",
>
> scnt,ib_status);
>
> return 1;
>
> }
>
> ++scnt;
>
> PERF_DEBUG("scnt = %d \n",scnt);
>
> }
>
>
>
> But no document just have to guess from the source code for what “qp”,
> “ctx” means and how to use them. These “ib_” initiated APIs were
> absolutely NOT WinSocket APIs. What we asked for is SDK or documents for
> how
> to using such API and which Header files we should include and which lib
> have to be linked. A sample project will be OK if no such document exists.
>
>
>
>
> Thanks
>
>
>
> Suo
>
> 2011/1/25 Smith, Stan <stan.smith at intel.com>
>
> Please submit modified question to ofw at lists.openfabrics.org
>
>
>
> Furthermore, please identify which sample code you are referring to in
> order that someone can respond to your question.
>
>
>
> Which Windows OS and processor (x86,x64,ia64).
>
>
>
> Generally speaking WSD (WinSock Direct) is not used directly but utilized
> by opening a socket(AF_INET) which beneath the socket layer that has a WSD
> provider enabled for the specific address family.
>
> For winOFED 2.3 release, run ‘installsp –l’ to see WSD providers.
>
> WSD in no longer enabled by default for svr2008* – see ‘installsp –I’
>
>
>
> Stan.
>
>
>
>
>
> From: *Suo Zhang* <zhangs at bjcarnation.com.cn>
> Date: 2011/1/20
> Subject: WSD Detail Sample Code
> To: membership at openfabrics.org
>
> Hi
>
> When I use WSD in my project, I found the sample code is very difficult to
> use. Because the sample code is very simple.
>
> I want to know where can I get a detail sample code for WSD. Because my
> application must use zero copy to get good performance.
>
> Thanks
>
> Best regards
>
>
>
> 张锁 Suo Zhang
>
> Sales Engineer
>
> 北京石竹科技有限公司
>
> Beijing Carnation Technologies Limited
>
> Mobile +86-186-1175-6167
>
> Tel：+8610-68587971-8029
>
> Fax：+8610-68587975
>
> Http://www.vme.cn <http://www.vme.cn/>
>
>
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ofw/attachments/20110127/a5ba92c3/attachment.html>
More information about the ofw
mailing list