[Openib-windows] File transfer performance options

[Fabian Tillier]

Hi Paul,

On 9/7/06, Paul Baxter <paul.baxter at dsl.pipex.com> wrote:
>
> My distributed Linux app reads multiple channels of sensor data at high
> speed in real time to local disks and at a convenient break switches from
> 'record' to 'archive' to allow a remote Windows archiver to remotely read
> recently recorded data and write to a much bigger Windows-based store.

Does your Linux app create several small files, or a few large files?
For several small files, SRP will probably be as good as
application-managed file transfers.  For large files you may be able
to get better bandwidth (close to wire speed) by managing the file
transfers in the application.  I would expect that with SRP you will
be able to reach close to 400MB/s.  Hopefully that will be acceptable.

> I think I may be describing things using the wrong terminology.

Nope, what you wrote makes sense - at least to me. :)

> Looking at this from the archiver point of view, I have a single
> windows-based application that is required to read from several quite
> separate Linux (RAID-based) volumes. Each volume is hosted within a separate
> Linux-based node.

Block level accesses should be no problem for your application's usage model.

> Can I have one windows-based SRP-initiator (or perhaps several instances of
> this on one windows node).

The OFW stack will discover each SRP target on the fabric and load an
instance of the SRP initiator miniport driver to service each target.

> One instance of an initiator deals exclusively with one Linux-based SRP
> client on one separate Linux node.
> [If multiple initiators are not possible, I can sequentially 'drain' each
> Linux disk and then close the SRP initiator and start a new one accessing a
> new Linux node (??)]

Multiple initiators will not be a problem as long as the disks they
connect to are distinct (which it sounds like they will be).

> The Linux nodes are 'serving' their data at the initiation of a
> windows-based client. (terminology?)

That's the correct terminology.

> I also appreciate that while Windows is remotely accessing a disk, the Linux
> machine must keep its hands off its local disk.

Theoretically, multiple hosts could read the data, as long as nobody writes.

> My Linux application does actually talk with the windows application to
> ensure exclusivity, I just didn't want my Linux app or the equivalent
> Windows side to actually have to transfer files via messages if remote file
> transfer semantics between drivers was sufficient.

Block level transfers should work fine for you.  Note however that you
might get better throughput using file transfers via messages due to
the ability to do huge RDMA requests.  For example, you could read a
1GB file into memory, register that memory, and do a single RDMA write
for the whole 1GB file into the remote system.  However, I suspect
that block level access via SRP will yield acceptable performance and
would require less custom software (assuming you can find an SRP
target driver for Linux).

> I'm hoping someone can now tell me that yes, SRP can do this today (or at a
> pinch tomorrow).

Yes, SRP can do what you want it to.  You will need an SRP target
driver for the Linux side, however.  Note that it doesn't matter what
IB drivers you use on the target as long as they behave properly on
the wire, so you if the stack that the SRP target reference code is
available and works, you could use that.

- Fab