[ofa-general] Installing nfs-rdma with OFED-1.4.1

Jon Mason jon at opengridcomputing.com
Mon Jul 13 13:40:49 PDT 2009 

On Fri, Jun 26, 2009 at 03:31:50PM +0100, Ross Smith wrote:
> Latest news:  I've bodged the installation, but I'm having problems
> with nfs.mount now.
> 
> Since everything worked with nfs-rdma on the development server, I
> manually copied the kerel-ib rpm file from there, and installed that
> on top of the default infiniband install on the live server, using the
> command:
> 
> # rpm -iv --replacepkgs kernel-ib.......
> 
> With that done modprobe xprtrdma now works.
> 
> Since the kernel-ib file appears to be the only one modified as
> nfsrdma support is added, I assume this is all that's needed.  Can
> anybody confirm if this is ok, or whether I'm going to break things
> horribly by doing this?
> 
> I also have nfs v1.1.6 installed, but it doesn't seem to understand
> the rdma option.  The system is still using tcp by default, and if I
> try to use rdma, I get this:
> 
> # mount -o rdma, port=20049 server:path /mnt
> mount.nfs: Unsupported nfs mount option: rdma
> 
> However, nfs is definitely the correct version:
> # mount.nfs -V
> mount.nfs (linux nfs-utils 1.1.6)
> 
> Even using the mount.rnfs utility supplied with OFED doesn't help:
> # rpm -iv rnfs-utils-1.1.5-2.OFED.i386.rpm
> # mount.rnfs server:path /mnt -o rdma,port=20049
> mount.rnfs: Unsupported nfs mount option: rdma

You need the "-i" option for older kernels.  In this case, you would want:
mount.rnfs server:path /mnt -i -o rdma,port=20049

> 
> And if I use mount.rnfs without the rdma option, "cat /proc/mounts"
> reports that it's just using tcp.
> 
> I'm getting read speeds of around 400MB/s using regular NFS which
> isn't bad, but I know I can achieve over 900MB/s with this server over
> Infiniband, so there's still some way to go.
> 
> Ross
> 
> 
> 
> On Fri, Jun 26, 2009 at 2:06 PM, Ross Smith<myxiplx at googlemail.com> wrote:
> > Ok, I've made a little more progress.
> >
> > The issue with nfs-utils was solved by installing efsprogs-devel, and
> > mount.nfs is now reporting that it's running v1.1.6.
> >
> > I still can't install OFED with nfs-rdma on the live server though, it
> > appears that it's unable to use the rpm package to install with nfs
> > support.
> >
> > I've now discovered how to use the ofed.conf file, and this is the
> > procedure I'm following:
> >
> > - On the development server, I run install.pl and perform a standard install
> > - Once that finishes, I edit the resulting ofed.conf and add nfsrdma=y
> > - Still on the development server I run:
> >  # install.pl -c ofed.conf
> > - That can be seen to install most items from the rpm, and to
> > recompile kernel-ib with the nfsrdma option.
> > - I then package up the OFED folder with tar, and send it to the live server
> > - On the live server I run the same command:
> >  # install.pl -c ofed.conf
> >
> > Despite it running with exactly the same options, I get the error:
> >
> > "/lib/modules/2.6.18......./build/scripts is required to build kernel-ib RPM.
> > Please install the corresponding kernel-source or kerenel-devel RPM."
> >
> > If I exclude the nfsrdma option and repeat all the above steps, the
> > script installs fine from the pre-compiled RPM.
> >
> > It looks to me like a problem with the install script when choosing nfs-rdma.
> >
> > Does anybody have any suggestions as to how I can work around this?
> > Am I able to simply install the RPM files manually?
> >
> > Ross
> >
> >
> >
> > On Fri, Jun 26, 2009 at 12:25 PM, Ross Smith<myxiplx at googlemail.com> wrote:
> >> Thanks Robert,
> >>
> >> Looking through your notes, we've followed similar steps.  I should
> >> mention that I do have Infiniband working on this server, including
> >> ipoib.  It's purely the nfs-rdma part I'm struggling with.
> >>
> >> My latest attempt has been to attempt a custom install, selecting
> >> nfs-rdma manually.  That worked fine on the development server, but
> >> when I attempted to run it on the live server I had an error:
> >>
> >> "/lib/modules/....../build/scripts is required to build kernel-ib RPM.
> >> Please install the corresponding kernel-source or kernel-devel RPM.
> >> tk rpm is required to install ibutils"
> >>
> >> The problem is, for XenServer, you can't get the kernel-source on the
> >> live server, instead Citrix ship a DDK (driver development kit)
> >> virtual machine, which is a pre-built system with all the kernel
> >> sources you need.
> >>
> >> OFED builds fine on the DDK machine, and for the default installation,
> >> I can compile everything there and then transfer the OFED folder (with
> >> the compiled RPM files) to the live server.  At that point install.pl
> >> runs fine, without needing to re-compile things.
> >>
> >> However, it appears that when I choose some elements in the custom
> >> install, it is not using these RPM files, and is trying to compile
> >> kernel-ib again from scratch.
> >>
> >> I have attached a file listing the custom modules I have chosen in
> >> case it helps.
> >>
> >> Ross
> >>
> >>
> >>
> >> On Fri, Jun 26, 2009 at 11:42 AM, Robert Dunkley<Robert at saq.co.uk> wrote:
> >>> Hi Ross,
> >>>
> >>> I installed OFED 1.3.1 (Older I know) with other options OK on a Centos
> >>> 5.2 Dom0 (Separate Xen install on Centos, not "Xen Server"), here is
> >>> some notes I took (I used the GUI installer)
> >>>
> >>> Hope this helps,
> >>>
> >>> Rob
> >>>
> >>> 1.      Download the OFED package from
> >>> http://www.openfabrics.org/downloads/OFED/
> >>> 2.      Extract it: tar -xvzf OFED-1.3.1.tgz
> >>> 3.      Check for the dependencies by running ./install.pl and choosing
> >>> documentation ("0").
> >>> 4.      Install all required dependencies using yum install package nam.
> >>> For Centos dependencies try: yum install gcc libstdc++-devel
> >>> libsysfs-devel tcl tcl-devel tk pciutils-devel kernel-devel rpm-build
> >>> gcc libtool bison flex tcl-devel swig gcc-c++ libtool kernel-xen-devel
> >>> zlib-devel You may need to run the yum command three times (It takes a
> >>> few runs for all the dependencies to sort themselves out)
> >>> 5.      It is recommended you now upgrade the kernel to make sure the
> >>> sources match it. For centos do: yum upgrade kernel (Or yum upgrade
> >>> kernel-xen for Xen setups)
> >>> 6.      Run ./install.pl again but this time choose to install ("2") and
> >>> choose custom install ("4").  See the appendix for a recommendation on
> >>> which modules to install.
> >>> 7.      Install should now complete, if it fails check the log (Most
> >>> likely you have not installed a required software package - go back to
> >>> step 3)
> >>> 8.      Say yes to configuring IPOIB (If you plan to use it) and
> >>> manually set the IP and subnet. When it asks for a network it wants the
> >>> base network address (If given an IP of 192.168.10.12 on a 255.255.255.0
> >>> subnet then the "Network" would be 192.168.10.0)
> >>> 9.      Reboot
> >>> 10.     Check the install using /etc/init.d/openibd status. This should
> >>> show one interface for each Infiniband port (First will be "ib0"). You
> >>> should also see quite a few modules loaded, the important ones are
> >>> (mlx4_ib & mlx4_core - QLogic drivers, mthca - Mellanox driver, ib_core
> >>> & ib_addr - core services, rdma_ucm & rdma_cm - RDMA, ib_cm & ib_ipoib -
> >>> IPOIB and ib_uverbs - Direct/Verb)
> >>> 11.     Run ibstat - This should display the installed devices and their
> >>> firmware versions and hardware / GUI IDs. If you need to update the
> >>> firmware for Mellanox see
> >>> http://www.mellanox.com/support/firmware_download.php I find it easier
> >>> to update using Windows and the WinOF software (Windows equivalent of
> >>> OFED - Available here: http://www.openfabrics.org/downloads/WinOF/ ).
> >>>
> >>> -----Original Message-----
> >>> From: general-bounces at lists.openfabrics.org
> >>> [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Ross Smith
> >>> Sent: 26 June 2009 11:36
> >>> To: nfs-rdma-devel at lists.sourceforge.net; general at lists.openfabrics.org
> >>> Cc: Yehonatan Yossef
> >>> Subject: [ofa-general] Installing nfs-rdma with OFED-1.4.1
> >>>
> >>> Hello everyone,
> >>>
> >>> Can anybody help me get nfs-rdma installed on CentOS 5.2 (actually
> >>> Citrix XenServer), with OFED-1.4.1?
> >>>
> >>> The default install of OFED doesn't include nfs-rdma, and I'm
> >>> struggling to work out which components I do and don't need for a
> >>> custom build.
> >>>
> >>> What I would like to include is:
> >>> - Core infiniband components
> >>> - Drivers for Mellanox Infinihost III Lx (MHES14)
> >>> - IPoIB
> >>> - nfs-rdma
> >>> - troubleshooting and diagnostic tools (ibstat, ibtracert, ibswitches,
> >>> ibhosts, etc...)
> >>>
> >>> My last attempt was following the instructions from OFED 1.4 - I
> >>> edited ofed.conf to add nfsrdma=y, and attempted a default install.
> >>> However after running install.pl, that change appears to have been
> >>> overwritten.
> >>>
> >>> I am also having problems updating mount.nfs on this machine.  Both
> >>> nfs-utils-1.1.6 and nfs-utils-1.2.0 fail to install with the error:
> >>> "blkid/blkid.h: No such file or directory", and I cannot yet find a
> >>> source for this file.
> >>>
> >>> Finally, my understanding is that nfs-rdma is in beta status for
> >>> CentOS 5.2, but I couldn't see any open bugs on the tracker, are there
> >>> any issues I should be aware of?
> >>>
> >>> thanks,
> >>>
> >>> Ross
> >>> _______________________________________________
> >>> general mailing list
> >>> general at lists.openfabrics.org
> >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> >>>
> >>> To unsubscribe, please visit
> >>> http://openib.org/mailman/listinfo/openib-general
> >>>
> >>> The SAQ Group
> >>>
> >>> Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ
> >>> SAQ is the trading name of SEMTEC Limited. Registered in England & Wales
> >>> Company Number: 06481952
> >>>
> >>> http://www.saqnet.co.uk AS29219
> >>>
> >>> SAQ Group Delivers high quality, honestly priced communication and I.T. services to UK Business.
> >>>
> >>> Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : Backups : Managed Networks : Remote Support.
> >>>
> >>> ISPA Member
> >>>
> >>>
> >>
> >
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

More information about the general mailing list