[Openib-windows] File transfer performance options

[Paul Baxter]

>
From: "Tzachi Dar" <tzachid at mellanox.co.il>
There is one thing that is missing from your mail, and that is if you
want to see the windows machine as some file server (for example SAMBA,
NFS, SRP), or are you ready to accept it as a normal server. The big
difference is that on the second option the server can be running at
user mode (for example FTP server).
<

The windows machine has to list and then choose amongst a set of files from
our Linux system and retrieve only relevant files e.g. those whose filename
relates to particular time slots.
We prefer not to write a Linux 'client' application to do this explicitly
but would rather have the windows machine's application access our data
files directly.
A few application-level locks are in place so that we won't be writing new
files to our local disks at the same time as the remote archiving accesses
them.

Other than that the main goal is to make the inter-OS (and inter-company)
interface as simple as possible. It currently doesn't seem that there is a
proven solution to support this at any transfer rate that takes significant
advantage of Infiniband.

I've specced my disks for 200 MB/s and we have DDR cards etc. (for other
reasons!), just no means to flex their muscles too easily using existing
COTS infrastructure.

>
When (the server application is) running at user mode, SDP can be used
as a socket provider.  This means that theoretically every socket
application should run and enjoy the speed of Infiniband. Currently
there are two projects of SDP under development: one is for Linux and
the other for Windows, so SDP can be used to allow machines from both
types to connect.
<

The key here is 'theoretical'. IMHO, Linux-Linux and Windows-Windows get a
lot more testing and priority than a Linux-Windows combination. (Which is
fair enough if that's where the market is.)

We've been burnt by this not being robustly tested and proven in reality in
cross-platform cases. (Note that this was before the current openfabrics
windows driver initiative).

>
Performance that we have measured on the windows
platform, using DDR cards was bigger than 1200 MB/Sec. (of course, this
data was from host memory, and not from disks).
<

We've used SDP previously in our Linux message interface and were very happy
with the results. Then someone included an old (v9 ) Solaris machine into
the mix so even before we tested on Windows, we ended up using
sockets/gigabit ethernet for command transfers.

SDP as an option for other parts of our application (large data transfers)
took a big turn for the worse when the previous Linux SDP implementation was
mothballed without a mature replacement. We've ended up writing our
application to use RDMA write directly now.

Note that I'm not too critical of the way SDP went away since I can
appreciate the need to greatly simplify the Linux SDP implementation, it did
leave people like me in the lurch however. I really appreciate the effort
put into these things by Michael Tsirkin et al. and look forward to the new
code in OFED 1.1


I'm also not sure that cross-platform operation of high-performance
Infiniband is near the top of anyone's agenda. Inside the windows world and
inside the Linux world things are looking rosey, but I'm largely stuck with
IPoIB or low-level verbs for cross-platform use.

SRP looks promising, but as a user, I see lots of statements that this SRP
initiator only works with that SRP target. Support for cross-platform
high-speed operation is 'coming soon'.

I'd love to know whether there has been significant testing between the
windows openfabrics  SRP initiator and the openIB Linux SRP target? Is this
on anyone's agenda. (Distinct from any windows SRP 'WHQL-certification
issues.) Is their even an 'inter-operable' standard that both
implementations can aspire to match?

>
So, if all you need to do is to pass files from one side to the other, I
would recommend that you will check this option.
<
Thanks for the tip. Maybe now the dust is settling on Linux SDP we may well
revisit this option.

>
One note about your experiments: when using ram disks, this probably
means that there is one more copy from the ram disk to the application
buffer. A real disk, has it's DMA engine, while a ram disk doesn't.
Another copy is probably not a problem when you are talking about
100MB/sec, but it would become a problem once you will use SDP (I hope).
<
We were only using these as a sanity check that physical disks weren't the
cause of the bottleneck.

>
Thanks
Tzachi
<
Thanks to you, Tzachi, and everyone helping to develop robust infiniband
support across a range of platforms.