[openib-general] RE: [Openib-windows] NFS performance and general disk network export advice (Linux-Windows)

[Fab Tillier]

Hi Paul,

> I'm looking to export a filesystem from each of four linux
> 64bit boxes to a single Windows server 2003 64bit Ed.
> 
> Has anyone achieved this already using an IB transport? Can
> I use NFS over IPoIB cross platform? i.e. do both ends
> support a solution?

IPoIB will interoperate cross platform, so any higher-level services you layer
above TCP/IP or UDP/IP should work fine.

> Is NFS over RDMA compatible with Windows (pretty sure the
> answer is no to this one but love to be proven wrong). I've
> attached Tom's announcement of the latest to the bottom of
> this email. I don't think Windows has the RDMA abstraction
> (yet)?

There is no NFS over RDMA file system for OpenIB Windows.  It would be great to
have it, but the focus is currently on getting the core stack stable and
released.

The long term goals, at least from my perspective, is to match functionality
between OpenIB Linux and Windows, even if the APIs aren't identical.  The
reality is that the iWARP crowd hasn't really been involved in the Windows
project, and have not driven any requirements, so that stack is continuing to be
focused on IB only.  I don't have a timeline for getting functionality matched
up, and we could certainly use more hands on deck for the Windows project.

> Are windows IB drivers (Openib or Mellanox) compatible with
> these options?
> Do I layer Windows services for Unix on top of the Windows IB
> drivers and IPoIB to achieve a cross platform NFS?

I don't know what you would need to do to get NFS working on Windows, but that
should be an orthogonal problem to getting IB working.  If NFS works on Windows
over GbE, it should work without a problem over IPoIB.

> Has anyone done much in the way of NFS performance
> comparisons of NFS over IPoIB in cross-platform situations
> vs say Gigabit ethernet. Does it work :) What is large file
> throughput and processor loading - I'm aiming for 150-200
> MB/s on large files on 4x SDR IB (possibly DDR if we can
> fit the bigger 144 port switch chassis into our rack layout
> for 50-ish nodes).

I can tell you that IPoIB performance on Windows is pretty awful.  The reason
for that is that the IPoIB driver shoehorns itself into the NDIS stack as a
802.3 Ethernet NIC, and thus gets 6-byte Ethernet MAC addresses.  Further,
Windows doesn't have any IB knowledge, so the IPoIB driver is responsible for
all ARP and DHCP encapsulation to match the IPoIB protocol on the wire.  This
involves snooping both outbound and inbound packets to see if they need
conversion, which does nasty stuff to performance.

Depending on the host CPU, 150-200MB/s should be achievable (I've seen 150+MB/s
in some of my testing).

> Are there any alternatives to using NFS that may be better
> and that would 'transparently' receive a performance boost
> with IB compared with using a simple NFS/gigabit ethernet
> solution. Must be fairly straightforward, ideally application
> neutral (configure a drive and load/unload script for Linux
> and it just happens) and compatible between Win2003 and
> Linux? Alternatives using perhaps Samba on the Linux side?

If you only have a single Windows box that has to read data from one or more
Linux boxes, you might have some success with making the Linux boxes SRP
targets, and then using the Windows SRP driver to access the Linux boxes.  The
SRP target driver would have to handle SRP commands and perform local disk
access.

Of course, the file system would have to be Windows compatible with this
solution, but you should be able to get the full RDMA performance since there
would be no network stack involved.  You'd also need to make sure that only a
single system accesses the data on the disks exported as SRP targets to prevent
corruption as those disks would appear as locally attached drives to the Windows
box.

I am unaware of an SRP target implementation for Linux, though, so that may not
be a viable option for you.

> My lack of knowledge of IB in the windows world has got me
> concerned over whether this is actually achievable (easily).
> 
> I hope to be trying this once we get a Windows 2003 machine,
> but hope someone can encourage me that its a breeze prior to
> my coming unstuck in a month or so!

The IB stuff should be a breeze to get functional and interoperating.  Whether
performance matches your requirements/expectations is another thing.  Do report
back if you have any questions or run into any problems along the way.

- Fab