[ofa-general] Installing nfs-rdma with OFED-1.4.1

Fri Jun 26 06:06:11 PDT 2009

Ok, I've made a little more progress.

The issue with nfs-utils was solved by installing efsprogs-devel, and
mount.nfs is now reporting that it's running v1.1.6.

I still can't install OFED with nfs-rdma on the live server though, it
appears that it's unable to use the rpm package to install with nfs
support.

I've now discovered how to use the ofed.conf file, and this is the
procedure I'm following:

- On the development server, I run install.pl and perform a standard install
- Once that finishes, I edit the resulting ofed.conf and add nfsrdma=y
- Still on the development server I run:
  # install.pl -c ofed.conf
- That can be seen to install most items from the rpm, and to
recompile kernel-ib with the nfsrdma option.
- I then package up the OFED folder with tar, and send it to the live server
- On the live server I run the same command:
  # install.pl -c ofed.conf

Despite it running with exactly the same options, I get the error:

"/lib/modules/2.6.18......./build/scripts is required to build kernel-ib RPM.
Please install the corresponding kernel-source or kerenel-devel RPM."

If I exclude the nfsrdma option and repeat all the above steps, the
script installs fine from the pre-compiled RPM.

It looks to me like a problem with the install script when choosing nfs-rdma.

Does anybody have any suggestions as to how I can work around this?
Am I able to simply install the RPM files manually?

Ross

On Fri, Jun 26, 2009 at 12:25 PM, Ross Smith<myxiplx at googlemail.com> wrote:
> Thanks Robert,
>
> Looking through your notes, we've followed similar steps.  I should
> mention that I do have Infiniband working on this server, including
> ipoib.  It's purely the nfs-rdma part I'm struggling with.
>
> My latest attempt has been to attempt a custom install, selecting
> nfs-rdma manually.  That worked fine on the development server, but
> when I attempted to run it on the live server I had an error:
>
> "/lib/modules/....../build/scripts is required to build kernel-ib RPM.
> Please install the corresponding kernel-source or kernel-devel RPM.
> tk rpm is required to install ibutils"
>
> The problem is, for XenServer, you can't get the kernel-source on the
> live server, instead Citrix ship a DDK (driver development kit)
> virtual machine, which is a pre-built system with all the kernel
> sources you need.
>
> OFED builds fine on the DDK machine, and for the default installation,
> I can compile everything there and then transfer the OFED folder (with
> the compiled RPM files) to the live server.  At that point install.pl
> runs fine, without needing to re-compile things.
>
> However, it appears that when I choose some elements in the custom
> install, it is not using these RPM files, and is trying to compile
> kernel-ib again from scratch.
>
> I have attached a file listing the custom modules I have chosen in
> case it helps.
>
> Ross
>
>
>
> On Fri, Jun 26, 2009 at 11:42 AM, Robert Dunkley<Robert at saq.co.uk> wrote:
>> Hi Ross,
>>
>> I installed OFED 1.3.1 (Older I know) with other options OK on a Centos
>> 5.2 Dom0 (Separate Xen install on Centos, not "Xen Server"), here is
>> some notes I took (I used the GUI installer)
>>
>> Hope this helps,
>>
>> Rob
>>
>> 1.      Download the OFED package from
>> http://www.openfabrics.org/downloads/OFED/
>> 2.      Extract it: tar -xvzf OFED-1.3.1.tgz
>> 3.      Check for the dependencies by running ./install.pl and choosing
>> documentation ("0").
>> 4.      Install all required dependencies using yum install package nam.
>> For Centos dependencies try: yum install gcc libstdc++-devel
>> libsysfs-devel tcl tcl-devel tk pciutils-devel kernel-devel rpm-build
>> gcc libtool bison flex tcl-devel swig gcc-c++ libtool kernel-xen-devel
>> zlib-devel You may need to run the yum command three times (It takes a
>> few runs for all the dependencies to sort themselves out)
>> 5.      It is recommended you now upgrade the kernel to make sure the
>> sources match it. For centos do: yum upgrade kernel (Or yum upgrade
>> kernel-xen for Xen setups)
>> 6.      Run ./install.pl again but this time choose to install ("2") and
>> choose custom install ("4").  See the appendix for a recommendation on
>> which modules to install.
>> 7.      Install should now complete, if it fails check the log (Most
>> likely you have not installed a required software package - go back to
>> step 3)
>> 8.      Say yes to configuring IPOIB (If you plan to use it) and
>> manually set the IP and subnet. When it asks for a network it wants the
>> base network address (If given an IP of 192.168.10.12 on a 255.255.255.0
>> subnet then the "Network" would be 192.168.10.0)
>> 9.      Reboot
>> 10.     Check the install using /etc/init.d/openibd status. This should
>> show one interface for each Infiniband port (First will be "ib0"). You
>> should also see quite a few modules loaded, the important ones are
>> (mlx4_ib & mlx4_core - QLogic drivers, mthca - Mellanox driver, ib_core
>> & ib_addr - core services, rdma_ucm & rdma_cm - RDMA, ib_cm & ib_ipoib -
>> IPOIB and ib_uverbs - Direct/Verb)
>> 11.     Run ibstat - This should display the installed devices and their
>> firmware versions and hardware / GUI IDs. If you need to update the
>> firmware for Mellanox see
>> http://www.mellanox.com/support/firmware_download.php I find it easier
>> to update using Windows and the WinOF software (Windows equivalent of
>> OFED - Available here: http://www.openfabrics.org/downloads/WinOF/ ).
>>
>> -----Original Message-----
>> From: general-bounces at lists.openfabrics.org
>> [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Ross Smith
>> Sent: 26 June 2009 11:36
>> To: nfs-rdma-devel at lists.sourceforge.net; general at lists.openfabrics.org
>> Cc: Yehonatan Yossef
>> Subject: [ofa-general] Installing nfs-rdma with OFED-1.4.1
>>
>> Hello everyone,
>>
>> Can anybody help me get nfs-rdma installed on CentOS 5.2 (actually
>> Citrix XenServer), with OFED-1.4.1?
>>
>> The default install of OFED doesn't include nfs-rdma, and I'm
>> struggling to work out which components I do and don't need for a
>> custom build.
>>
>> What I would like to include is:
>> - Core infiniband components
>> - Drivers for Mellanox Infinihost III Lx (MHES14)
>> - IPoIB
>> - nfs-rdma
>> - troubleshooting and diagnostic tools (ibstat, ibtracert, ibswitches,
>> ibhosts, etc...)
>>
>> My last attempt was following the instructions from OFED 1.4 - I
>> edited ofed.conf to add nfsrdma=y, and attempted a default install.
>> However after running install.pl, that change appears to have been
>> overwritten.
>>
>> I am also having problems updating mount.nfs on this machine.  Both
>> nfs-utils-1.1.6 and nfs-utils-1.2.0 fail to install with the error:
>> "blkid/blkid.h: No such file or directory", and I cannot yet find a
>> source for this file.
>>
>> Finally, my understanding is that nfs-rdma is in beta status for
>> CentOS 5.2, but I couldn't see any open bugs on the tracker, are there
>> any issues I should be aware of?
>>
>> thanks,
>>
>> Ross
>> _______________________________________________
>> general mailing list
>> general at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>
>> To unsubscribe, please visit
>> http://openib.org/mailman/listinfo/openib-general
>>
>> The SAQ Group
>>
>> Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ
>> SAQ is the trading name of SEMTEC Limited. Registered in England & Wales
>> Company Number: 06481952
>>
>> http://www.saqnet.co.uk AS29219
>>
>> SAQ Group Delivers high quality, honestly priced communication and I.T. services to UK Business.
>>
>> Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : Backups : Managed Networks : Remote Support.
>>
>> ISPA Member
>>
>>
>