[ofa-general] Installing nfs-rdma with OFED-1.4.1

Ross Smith myxiplx at googlemail.com
Fri Jun 26 11:03:14 PDT 2009 

Hi Jeff,

Yeah, I tried that, but while it seemed to work ok on the server with
the development tools installed, it didn't work on the live server.
The install script kept trying to recompile the kernel-ib package
which isn't possible on the live environment.

Copying the pre-compiled kernel-ip package from the development to the
live server, and manually installing that seemed to work, but like I
said above, the problem now seems to be with the mount.nfs and
mount.rnfs commands - neither understood the rdma option.

Ross



On Fri, Jun 26, 2009 at 6:15 PM, Jeff Becker<Jeffrey.C.Becker at nasa.gov> wrote:
> Hi Ross. The easiest way to install NFSRDMA is to do a custom install
> and select it manually from the kernel-ib modules. Also be sure to
> select rnfs-utils since that includes a special mount.rnfs that is
> needed to understand RDMA mounts.
>
> -jeff
>
> Ross Smith wrote:
>> Latest news:  I've bodged the installation, but I'm having problems
>> with nfs.mount now.
>>
>> Since everything worked with nfs-rdma on the development server, I
>> manually copied the kerel-ib rpm file from there, and installed that
>> on top of the default infiniband install on the live server, using the
>> command:
>>
>> # rpm -iv --replacepkgs kernel-ib.......
>>
>> With that done modprobe xprtrdma now works.
>>
>> Since the kernel-ib file appears to be the only one modified as
>> nfsrdma support is added, I assume this is all that's needed.  Can
>> anybody confirm if this is ok, or whether I'm going to break things
>> horribly by doing this?
>>
>> I also have nfs v1.1.6 installed, but it doesn't seem to understand
>> the rdma option.  The system is still using tcp by default, and if I
>> try to use rdma, I get this:
>>
>> # mount -o rdma, port=20049 server:path /mnt
>> mount.nfs: Unsupported nfs mount option: rdma
>>
>> However, nfs is definitely the correct version:
>> # mount.nfs -V
>> mount.nfs (linux nfs-utils 1.1.6)
>>
>> Even using the mount.rnfs utility supplied with OFED doesn't help:
>> # rpm -iv rnfs-utils-1.1.5-2.OFED.i386.rpm
>> # mount.rnfs server:path /mnt -o rdma,port=20049
>> mount.rnfs: Unsupported nfs mount option: rdma
>>
>> And if I use mount.rnfs without the rdma option, "cat /proc/mounts"
>> reports that it's just using tcp.
>>
>> I'm getting read speeds of around 400MB/s using regular NFS which
>> isn't bad, but I know I can achieve over 900MB/s with this server over
>> Infiniband, so there's still some way to go.
>>
>> Ross
>>
>>
>>
>> On Fri, Jun 26, 2009 at 2:06 PM, Ross Smith<myxiplx at googlemail.com> wrote:
>>
>>> Ok, I've made a little more progress.
>>>
>>> The issue with nfs-utils was solved by installing efsprogs-devel, and
>>> mount.nfs is now reporting that it's running v1.1.6.
>>>
>>> I still can't install OFED with nfs-rdma on the live server though, it
>>> appears that it's unable to use the rpm package to install with nfs
>>> support.
>>>
>>> I've now discovered how to use the ofed.conf file, and this is the
>>> procedure I'm following:
>>>
>>> - On the development server, I run install.pl and perform a standard install
>>> - Once that finishes, I edit the resulting ofed.conf and add nfsrdma=y
>>> - Still on the development server I run:
>>>  # install.pl -c ofed.conf
>>> - That can be seen to install most items from the rpm, and to
>>> recompile kernel-ib with the nfsrdma option.
>>> - I then package up the OFED folder with tar, and send it to the live server
>>> - On the live server I run the same command:
>>>  # install.pl -c ofed.conf
>>>
>>> Despite it running with exactly the same options, I get the error:
>>>
>>> "/lib/modules/2.6.18......./build/scripts is required to build kernel-ib RPM.
>>> Please install the corresponding kernel-source or kerenel-devel RPM."
>>>
>>> If I exclude the nfsrdma option and repeat all the above steps, the
>>> script installs fine from the pre-compiled RPM.
>>>
>>> It looks to me like a problem with the install script when choosing nfs-rdma.
>>>
>>> Does anybody have any suggestions as to how I can work around this?
>>> Am I able to simply install the RPM files manually?
>>>
>>> Ross
>>>
>>>
>>>
>>> On Fri, Jun 26, 2009 at 12:25 PM, Ross Smith<myxiplx at googlemail.com> wrote:
>>>
>>>> Thanks Robert,
>>>>
>>>> Looking through your notes, we've followed similar steps.  I should
>>>> mention that I do have Infiniband working on this server, including
>>>> ipoib.  It's purely the nfs-rdma part I'm struggling with.
>>>>
>>>> My latest attempt has been to attempt a custom install, selecting
>>>> nfs-rdma manually.  That worked fine on the development server, but
>>>> when I attempted to run it on the live server I had an error:
>>>>
>>>> "/lib/modules/....../build/scripts is required to build kernel-ib RPM.
>>>> Please install the corresponding kernel-source or kernel-devel RPM.
>>>> tk rpm is required to install ibutils"
>>>>
>>>> The problem is, for XenServer, you can't get the kernel-source on the
>>>> live server, instead Citrix ship a DDK (driver development kit)
>>>> virtual machine, which is a pre-built system with all the kernel
>>>> sources you need.
>>>>
>>>> OFED builds fine on the DDK machine, and for the default installation,
>>>> I can compile everything there and then transfer the OFED folder (with
>>>> the compiled RPM files) to the live server.  At that point install.pl
>>>> runs fine, without needing to re-compile things.
>>>>
>>>> However, it appears that when I choose some elements in the custom
>>>> install, it is not using these RPM files, and is trying to compile
>>>> kernel-ib again from scratch.
>>>>
>>>> I have attached a file listing the custom modules I have chosen in
>>>> case it helps.
>>>>
>>>> Ross
>>>>
>>>>
>>>>
>>>> On Fri, Jun 26, 2009 at 11:42 AM, Robert Dunkley<Robert at saq.co.uk> wrote:
>>>>
>>>>> Hi Ross,
>>>>>
>>>>> I installed OFED 1.3.1 (Older I know) with other options OK on a Centos
>>>>> 5.2 Dom0 (Separate Xen install on Centos, not "Xen Server"), here is
>>>>> some notes I took (I used the GUI installer)
>>>>>
>>>>> Hope this helps,
>>>>>
>>>>> Rob
>>>>>
>>>>> 1.      Download the OFED package from
>>>>> http://www.openfabrics.org/downloads/OFED/
>>>>> 2.      Extract it: tar -xvzf OFED-1.3.1.tgz
>>>>> 3.      Check for the dependencies by running ./install.pl and choosing
>>>>> documentation ("0").
>>>>> 4.      Install all required dependencies using yum install package nam.
>>>>> For Centos dependencies try: yum install gcc libstdc++-devel
>>>>> libsysfs-devel tcl tcl-devel tk pciutils-devel kernel-devel rpm-build
>>>>> gcc libtool bison flex tcl-devel swig gcc-c++ libtool kernel-xen-devel
>>>>> zlib-devel You may need to run the yum command three times (It takes a
>>>>> few runs for all the dependencies to sort themselves out)
>>>>> 5.      It is recommended you now upgrade the kernel to make sure the
>>>>> sources match it. For centos do: yum upgrade kernel (Or yum upgrade
>>>>> kernel-xen for Xen setups)
>>>>> 6.      Run ./install.pl again but this time choose to install ("2") and
>>>>> choose custom install ("4").  See the appendix for a recommendation on
>>>>> which modules to install.
>>>>> 7.      Install should now complete, if it fails check the log (Most
>>>>> likely you have not installed a required software package - go back to
>>>>> step 3)
>>>>> 8.      Say yes to configuring IPOIB (If you plan to use it) and
>>>>> manually set the IP and subnet. When it asks for a network it wants the
>>>>> base network address (If given an IP of 192.168.10.12 on a 255.255.255.0
>>>>> subnet then the "Network" would be 192.168.10.0)
>>>>> 9.      Reboot
>>>>> 10.     Check the install using /etc/init.d/openibd status. This should
>>>>> show one interface for each Infiniband port (First will be "ib0"). You
>>>>> should also see quite a few modules loaded, the important ones are
>>>>> (mlx4_ib & mlx4_core - QLogic drivers, mthca - Mellanox driver, ib_core
>>>>> & ib_addr - core services, rdma_ucm & rdma_cm - RDMA, ib_cm & ib_ipoib -
>>>>> IPOIB and ib_uverbs - Direct/Verb)
>>>>> 11.     Run ibstat - This should display the installed devices and their
>>>>> firmware versions and hardware / GUI IDs. If you need to update the
>>>>> firmware for Mellanox see
>>>>> http://www.mellanox.com/support/firmware_download.php I find it easier
>>>>> to update using Windows and the WinOF software (Windows equivalent of
>>>>> OFED - Available here: http://www.openfabrics.org/downloads/WinOF/ ).
>>>>>
>>>>> -----Original Message-----
>>>>> From: general-bounces at lists.openfabrics.org
>>>>> [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Ross Smith
>>>>> Sent: 26 June 2009 11:36
>>>>> To: nfs-rdma-devel at lists.sourceforge.net; general at lists.openfabrics.org
>>>>> Cc: Yehonatan Yossef
>>>>> Subject: [ofa-general] Installing nfs-rdma with OFED-1.4.1
>>>>>
>>>>> Hello everyone,
>>>>>
>>>>> Can anybody help me get nfs-rdma installed on CentOS 5.2 (actually
>>>>> Citrix XenServer), with OFED-1.4.1?
>>>>>
>>>>> The default install of OFED doesn't include nfs-rdma, and I'm
>>>>> struggling to work out which components I do and don't need for a
>>>>> custom build.
>>>>>
>>>>> What I would like to include is:
>>>>> - Core infiniband components
>>>>> - Drivers for Mellanox Infinihost III Lx (MHES14)
>>>>> - IPoIB
>>>>> - nfs-rdma
>>>>> - troubleshooting and diagnostic tools (ibstat, ibtracert, ibswitches,
>>>>> ibhosts, etc...)
>>>>>
>>>>> My last attempt was following the instructions from OFED 1.4 - I
>>>>> edited ofed.conf to add nfsrdma=y, and attempted a default install.
>>>>> However after running install.pl, that change appears to have been
>>>>> overwritten.
>>>>>
>>>>> I am also having problems updating mount.nfs on this machine.  Both
>>>>> nfs-utils-1.1.6 and nfs-utils-1.2.0 fail to install with the error:
>>>>> "blkid/blkid.h: No such file or directory", and I cannot yet find a
>>>>> source for this file.
>>>>>
>>>>> Finally, my understanding is that nfs-rdma is in beta status for
>>>>> CentOS 5.2, but I couldn't see any open bugs on the tracker, are there
>>>>> any issues I should be aware of?
>>>>>
>>>>> thanks,
>>>>>
>>>>> Ross
>>>>> _______________________________________________
>>>>> general mailing list
>>>>> general at lists.openfabrics.org
>>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>>>
>>>>> To unsubscribe, please visit
>>>>> http://openib.org/mailman/listinfo/openib-general
>>>>>
>>>>> The SAQ Group
>>>>>
>>>>> Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ
>>>>> SAQ is the trading name of SEMTEC Limited. Registered in England & Wales
>>>>> Company Number: 06481952
>>>>>
>>>>> http://www.saqnet.co.uk AS29219
>>>>>
>>>>> SAQ Group Delivers high quality, honestly priced communication and I.T. services to UK Business.
>>>>>
>>>>> Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : Backups : Managed Networks : Remote Support.
>>>>>
>>>>> ISPA Member
>>>>>
>>>>>
>>>>>
>> _______________________________________________
>> general mailing list
>> general at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>
>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>>
>
>

More information about the general mailing list