[openib-general] RE: [Openib-windows] NFS performance and general disk network export advice (Linux-Windows)

[Tom Tucker]

Fab:

As you point out, we've been focused on the main trunk and or target
test platforms are Linux based. That said, we actually had Beta versions
of NDIS and Winsock Direct drivers for the AMSO adapter, so we know this
works and we know where the dead are buried.

It probably makes sense to wait until the core iWARP support is merged
into the main trunk. however, when/if you decide to merge up from the
main trunk and pick up iWARP support, I am more than happy to help you
with any issues that you may have.

Tom


On Thu, 2006-02-09 at 14:02 -0800, Fab Tillier wrote:
> Hi Paul,
> 
> > I'm looking to export a filesystem from each of four linux
> > 64bit boxes to a single Windows server 2003 64bit Ed.
> > 
> > Has anyone achieved this already using an IB transport? Can
> > I use NFS over IPoIB cross platform? i.e. do both ends
> > support a solution?
> 
> IPoIB will interoperate cross platform, so any higher-level services you layer
> above TCP/IP or UDP/IP should work fine.
> 
> > Is NFS over RDMA compatible with Windows (pretty sure the
> > answer is no to this one but love to be proven wrong). I've
> > attached Tom's announcement of the latest to the bottom of
> > this email. I don't think Windows has the RDMA abstraction
> > (yet)?
> 
> There is no NFS over RDMA file system for OpenIB Windows.  It would be great to
> have it, but the focus is currently on getting the core stack stable and
> released.
> 
> The long term goals, at least from my perspective, is to match functionality
> between OpenIB Linux and Windows, even if the APIs aren't identical.  The
> reality is that the iWARP crowd hasn't really been involved in the Windows
> project, and have not driven any requirements, so that stack is continuing to be
> focused on IB only.  I don't have a timeline for getting functionality matched
> up, and we could certainly use more hands on deck for the Windows project.
> 
> > Are windows IB drivers (Openib or Mellanox) compatible with
> > these options?
> > Do I layer Windows services for Unix on top of the Windows IB
> > drivers and IPoIB to achieve a cross platform NFS?
> 
> I don't know what you would need to do to get NFS working on Windows, but that
> should be an orthogonal problem to getting IB working.  If NFS works on Windows
> over GbE, it should work without a problem over IPoIB.
> 
> > Has anyone done much in the way of NFS performance
> > comparisons of NFS over IPoIB in cross-platform situations
> > vs say Gigabit ethernet. Does it work :) What is large file
> > throughput and processor loading - I'm aiming for 150-200
> > MB/s on large files on 4x SDR IB (possibly DDR if we can
> > fit the bigger 144 port switch chassis into our rack layout
> > for 50-ish nodes).
> 
> I can tell you that IPoIB performance on Windows is pretty awful.  The reason
> for that is that the IPoIB driver shoehorns itself into the NDIS stack as a
> 802.3 Ethernet NIC, and thus gets 6-byte Ethernet MAC addresses.  Further,
> Windows doesn't have any IB knowledge, so the IPoIB driver is responsible for
> all ARP and DHCP encapsulation to match the IPoIB protocol on the wire.  This
> involves snooping both outbound and inbound packets to see if they need
> conversion, which does nasty stuff to performance.
> 
> Depending on the host CPU, 150-200MB/s should be achievable (I've seen 150+MB/s
> in some of my testing).
> 
> > Are there any alternatives to using NFS that may be better
> > and that would 'transparently' receive a performance boost
> > with IB compared with using a simple NFS/gigabit ethernet
> > solution. Must be fairly straightforward, ideally application
> > neutral (configure a drive and load/unload script for Linux
> > and it just happens) and compatible between Win2003 and
> > Linux? Alternatives using perhaps Samba on the Linux side?
> 
> If you only have a single Windows box that has to read data from one or more
> Linux boxes, you might have some success with making the Linux boxes SRP
> targets, and then using the Windows SRP driver to access the Linux boxes.  The
> SRP target driver would have to handle SRP commands and perform local disk
> access.
> 
> Of course, the file system would have to be Windows compatible with this
> solution, but you should be able to get the full RDMA performance since there
> would be no network stack involved.  You'd also need to make sure that only a
> single system accesses the data on the disks exported as SRP targets to prevent
> corruption as those disks would appear as locally attached drives to the Windows
> box.
> 
> I am unaware of an SRP target implementation for Linux, though, so that may not
> be a viable option for you.
> 
> > My lack of knowledge of IB in the windows world has got me
> > concerned over whether this is actually achievable (easily).
> > 
> > I hope to be trying this once we get a Windows 2003 machine,
> > but hope someone can encourage me that its a breeze prior to
> > my coming unstuck in a month or so!
> 
> The IB stuff should be a breeze to get functional and interoperating.  Whether
> performance matches your requirements/expectations is another thing.  Do report
> back if you have any questions or run into any problems along the way.
> 
> - Fab
> 
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general