[Openib-windows] File transfer performance options

[Tzachi Dar]

Hi Paul,

In the beginning of this mail thread you have described a problem of
passing files from a Linux server to windows server. You have described
many experiments that you did and the fact that the performance that you
received was not as good as expected.

In reply I have advised you to consider using SDP for this file
transfers. if to summarize your answer in one sentence you said that SDP
is still not ready. I would have loved to tell you that SDP is ready,
but unfortunately the windows SDP is not a product yet. However, it is
mature enough to start doing some measurements which is what I did.

I have changed a simple benchmark program that I had to also write it's
data to disk. As a disk I have used AMT Ramdisk (512 MB). I have run two
instances of this program, and got the results of 578 MB/sec which is
considerably higher than results that you have achieved using other
experiments. (one client gave me 450 MB/sec)
Please note that since data is being copied 3 times in this scenario, we
are standing near the theoretical speed of the machine (one copy is from
the HCA to the kernel buffer, another is from the kernel buffer to the
application buffer, and that last copy is from the application buffer to
the Ram Disk).

It is true that the development road of your application might force you
not to use SDP, as SDP is not in production right now, but if you can
wait the extra time than please note that SDP can supply the BW.

Thanks
Tzachi

> -----Original Message-----
> From: Paul Baxter [mailto:paul.baxter at dsl.pipex.com] 
> Sent: Friday, September 01, 2006 1:11 AM
> To: openib-windows at openib.org; Tzachi Dar
> Subject: Re: [Openib-windows] File transfer performance options
> 
> >
> From: "Tzachi Dar" <tzachid at mellanox.co.il> There is one 
> thing that is missing from your mail, and that is if you want 
> to see the windows machine as some file server (for example 
> SAMBA, NFS, SRP), or are you ready to accept it as a normal 
> server. The big difference is that on the second option the 
> server can be running at user mode (for example FTP server).
> <
> 
> The windows machine has to list and then choose amongst a set 
> of files from our Linux system and retrieve only relevant 
> files e.g. those whose filename relates to particular time slots.
> We prefer not to write a Linux 'client' application to do 
> this explicitly but would rather have the windows machine's 
> application access our data files directly.
> A few application-level locks are in place so that we won't 
> be writing new files to our local disks at the same time as 
> the remote archiving accesses them.
> 
> Other than that the main goal is to make the inter-OS (and 
> inter-company) interface as simple as possible. It currently 
> doesn't seem that there is a proven solution to support this 
> at any transfer rate that takes significant advantage of Infiniband.
> 
> I've specced my disks for 200 MB/s and we have DDR cards etc. 
> (for other reasons!), just no means to flex their muscles too 
> easily using existing COTS infrastructure.
> 
> >
> When (the server application is) running at user mode, SDP 
> can be used as a socket provider.  This means that 
> theoretically every socket application should run and enjoy 
> the speed of Infiniband. Currently there are two projects of 
> SDP under development: one is for Linux and the other for 
> Windows, so SDP can be used to allow machines from both types 
> to connect.
> <
> 
> The key here is 'theoretical'. IMHO, Linux-Linux and 
> Windows-Windows get a lot more testing and priority than a 
> Linux-Windows combination. (Which is fair enough if that's 
> where the market is.)
> 
> We've been burnt by this not being robustly tested and proven 
> in reality in cross-platform cases. (Note that this was 
> before the current openfabrics windows driver initiative).
> 
> >
> Performance that we have measured on the windows platform, 
> using DDR cards was bigger than 1200 MB/Sec. (of course, this 
> data was from host memory, and not from disks).
> <
> 
> We've used SDP previously in our Linux message interface and 
> were very happy with the results. Then someone included an 
> old (v9 ) Solaris machine into the mix so even before we 
> tested on Windows, we ended up using sockets/gigabit ethernet 
> for command transfers.
> 
> SDP as an option for other parts of our application (large 
> data transfers) took a big turn for the worse when the 
> previous Linux SDP implementation was mothballed without a 
> mature replacement. We've ended up writing our application to 
> use RDMA write directly now.
> 
> Note that I'm not too critical of the way SDP went away since 
> I can appreciate the need to greatly simplify the Linux SDP 
> implementation, it did leave people like me in the lurch 
> however. I really appreciate the effort put into these things 
> by Michael Tsirkin et al. and look forward to the new code in OFED 1.1
> 
> 
> I'm also not sure that cross-platform operation of 
> high-performance Infiniband is near the top of anyone's 
> agenda. Inside the windows world and inside the Linux world 
> things are looking rosey, but I'm largely stuck with IPoIB or 
> low-level verbs for cross-platform use.
> 
> SRP looks promising, but as a user, I see lots of statements 
> that this SRP initiator only works with that SRP target. 
> Support for cross-platform high-speed operation is 'coming soon'.
> 
> I'd love to know whether there has been significant testing 
> between the windows openfabrics  SRP initiator and the openIB 
> Linux SRP target? Is this on anyone's agenda. (Distinct from 
> any windows SRP 'WHQL-certification
> issues.) Is their even an 'inter-operable' standard that both 
> implementations can aspire to match?
> 
> >
> So, if all you need to do is to pass files from one side to 
> the other, I would recommend that you will check this option.
> <
> Thanks for the tip. Maybe now the dust is settling on Linux 
> SDP we may well revisit this option.
> 
> >
> One note about your experiments: when using ram disks, this 
> probably means that there is one more copy from the ram disk 
> to the application buffer. A real disk, has it's DMA engine, 
> while a ram disk doesn't.
> Another copy is probably not a problem when you are talking 
> about 100MB/sec, but it would become a problem once you will 
> use SDP (I hope).
> <
> We were only using these as a sanity check that physical 
> disks weren't the cause of the bottleneck.
> 
> >
> Thanks
> Tzachi
> <
> Thanks to you, Tzachi, and everyone helping to develop robust 
> infiniband support across a range of platforms.
>