[ofa-general] Installing nfs-rdma with OFED-1.4.1

[Ross Smith]

Thanks Robert,

Looking through your notes, we've followed similar steps.  I should
mention that I do have Infiniband working on this server, including
ipoib.  It's purely the nfs-rdma part I'm struggling with.

My latest attempt has been to attempt a custom install, selecting
nfs-rdma manually.  That worked fine on the development server, but
when I attempted to run it on the live server I had an error:

"/lib/modules/....../build/scripts is required to build kernel-ib RPM.
Please install the corresponding kernel-source or kernel-devel RPM.
tk rpm is required to install ibutils"

The problem is, for XenServer, you can't get the kernel-source on the
live server, instead Citrix ship a DDK (driver development kit)
virtual machine, which is a pre-built system with all the kernel
sources you need.

OFED builds fine on the DDK machine, and for the default installation,
I can compile everything there and then transfer the OFED folder (with
the compiled RPM files) to the live server.  At that point install.pl
runs fine, without needing to re-compile things.

However, it appears that when I choose some elements in the custom
install, it is not using these RPM files, and is trying to compile
kernel-ib again from scratch.

I have attached a file listing the custom modules I have chosen in
case it helps.

Ross



On Fri, Jun 26, 2009 at 11:42 AM, Robert Dunkley<Robert at saq.co.uk> wrote:
> Hi Ross,
>
> I installed OFED 1.3.1 (Older I know) with other options OK on a Centos
> 5.2 Dom0 (Separate Xen install on Centos, not "Xen Server"), here is
> some notes I took (I used the GUI installer)
>
> Hope this helps,
>
> Rob
>
> 1.      Download the OFED package from
> http://www.openfabrics.org/downloads/OFED/
> 2.      Extract it: tar -xvzf OFED-1.3.1.tgz
> 3.      Check for the dependencies by running ./install.pl and choosing
> documentation ("0").
> 4.      Install all required dependencies using yum install package nam.
> For Centos dependencies try: yum install gcc libstdc++-devel
> libsysfs-devel tcl tcl-devel tk pciutils-devel kernel-devel rpm-build
> gcc libtool bison flex tcl-devel swig gcc-c++ libtool kernel-xen-devel
> zlib-devel You may need to run the yum command three times (It takes a
> few runs for all the dependencies to sort themselves out)
> 5.      It is recommended you now upgrade the kernel to make sure the
> sources match it. For centos do: yum upgrade kernel (Or yum upgrade
> kernel-xen for Xen setups)
> 6.      Run ./install.pl again but this time choose to install ("2") and
> choose custom install ("4").  See the appendix for a recommendation on
> which modules to install.
> 7.      Install should now complete, if it fails check the log (Most
> likely you have not installed a required software package - go back to
> step 3)
> 8.      Say yes to configuring IPOIB (If you plan to use it) and
> manually set the IP and subnet. When it asks for a network it wants the
> base network address (If given an IP of 192.168.10.12 on a 255.255.255.0
> subnet then the "Network" would be 192.168.10.0)
> 9.      Reboot
> 10.     Check the install using /etc/init.d/openibd status. This should
> show one interface for each Infiniband port (First will be "ib0"). You
> should also see quite a few modules loaded, the important ones are
> (mlx4_ib & mlx4_core - QLogic drivers, mthca - Mellanox driver, ib_core
> & ib_addr - core services, rdma_ucm & rdma_cm - RDMA, ib_cm & ib_ipoib -
> IPOIB and ib_uverbs - Direct/Verb)
> 11.     Run ibstat - This should display the installed devices and their
> firmware versions and hardware / GUI IDs. If you need to update the
> firmware for Mellanox see
> http://www.mellanox.com/support/firmware_download.php I find it easier
> to update using Windows and the WinOF software (Windows equivalent of
> OFED - Available here: http://www.openfabrics.org/downloads/WinOF/ ).
>
> -----Original Message-----
> From: general-bounces at lists.openfabrics.org
> [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Ross Smith
> Sent: 26 June 2009 11:36
> To: nfs-rdma-devel at lists.sourceforge.net; general at lists.openfabrics.org
> Cc: Yehonatan Yossef
> Subject: [ofa-general] Installing nfs-rdma with OFED-1.4.1
>
> Hello everyone,
>
> Can anybody help me get nfs-rdma installed on CentOS 5.2 (actually
> Citrix XenServer), with OFED-1.4.1?
>
> The default install of OFED doesn't include nfs-rdma, and I'm
> struggling to work out which components I do and don't need for a
> custom build.
>
> What I would like to include is:
> - Core infiniband components
> - Drivers for Mellanox Infinihost III Lx (MHES14)
> - IPoIB
> - nfs-rdma
> - troubleshooting and diagnostic tools (ibstat, ibtracert, ibswitches,
> ibhosts, etc...)
>
> My last attempt was following the instructions from OFED 1.4 - I
> edited ofed.conf to add nfsrdma=y, and attempted a default install.
> However after running install.pl, that change appears to have been
> overwritten.
>
> I am also having problems updating mount.nfs on this machine.  Both
> nfs-utils-1.1.6 and nfs-utils-1.2.0 fail to install with the error:
> "blkid/blkid.h: No such file or directory", and I cannot yet find a
> source for this file.
>
> Finally, my understanding is that nfs-rdma is in beta status for
> CentOS 5.2, but I couldn't see any open bugs on the tracker, are there
> any issues I should be aware of?
>
> thanks,
>
> Ross
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit
> http://openib.org/mailman/listinfo/openib-general
>
> The SAQ Group
>
> Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ
> SAQ is the trading name of SEMTEC Limited. Registered in England & Wales
> Company Number: 06481952
>
> http://www.saqnet.co.uk AS29219
>
> SAQ Group Delivers high quality, honestly priced communication and I.T. services to UK Business.
>
> Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : Backups : Managed Networks : Remote Support.
>
> ISPA Member
>
>
-------------- next part --------------
packages to install:


Select Option [1-4]:4
Install kernel-ib? [y/N]:y
Install core module? [y/N]:y
Install mthca module? [y/N]:y
Install mlx4 module? [y/N]:n
Install mlx4_en module? [y/N]:n
Install cxgb3 module? [y/N]:n
Install nes module? [y/N]:n
Install ipoib module? [y/N]:y
Install sdp module? [y/N]:n
Install srp module? [y/N]:n
Install srpt module? [y/N]:n
Install rds module? [y/N]:n
Install qlgc_vnic module? [y/N]:n
Install iser module? (open-iscsi will also be installed) [y/N]:n
Install nfsrdma module? [y/N]:y
Install kernel-ib-devel? [y/N]:n
Install ib-bonding? [y/N]:n
Install ib-bonding-debuginfo? [y/N]:n
Install libibverbs? [y/N]:y
Install libibverbs-devel? [y/N]:y
Install libibverbs-devel-static? [y/N]:n
Install libibverbs-utils? [y/N]:y
Install libibverbs-debuginfo? [y/N]:n
Install libmthca? [y/N]:y
Install libmthca-devel-static? [y/N]:n
Install libmthca-debuginfo? [y/N]:n
Install libmlx4? [y/N]:n
Install libmlx4-devel? [y/N]:n
Install libmlx4-debuginfo? [y/N]:n
Install libcxgb3? [y/N]:n
Install libcxgb3-devel? [y/N]:n
Install libcxgb3-debuginfo? [y/N]:n
Install libnes? [y/N]:n
Install libnes-devel-static? [y/N]:n
Install libnes-debuginfo? [y/N]:n
Install libibcm? [y/N]:n
Install libibcm-devel? [y/N]:n
Install libibcm-debuginfo? [y/N]:n
Install libibcommon? [y/N]:y
Install libibcommon-devel? [y/N]:n
Install libibcommon-static? [y/N]:n
Install libibcommon-debuginfo? [y/N]:n
Install libibumad? [y/N]:y
Install libibumad-devel? [y/N]:n
Install libibumad-static? [y/N]:n
Install libibumad-debuginfo? [y/N]:n
Install libibmad? [y/N]:n
Install libibmad-devel? [y/N]:n
Install libibmad-static? [y/N]:n
Install libibmad-debuginfo? [y/N]:n
Install ibsim? [y/N]:n
Install ibsim-debuginfo? [y/N]:n
Install librdmacm? [y/N]:y
Install librdmacm-utils? [y/N]:y
Install librdmacm-devel? [y/N]:n
Install librdmacm-debuginfo? [y/N]:n
Install libsdp? [y/N]:n
Install libsdp-devel? [y/N]:n
Install libsdp-debuginfo? [y/N]:n
Install opensm? [y/N]:n
Install opensm-libs? [y/N]:n
Install opensm-devel? [y/N]:n
Install opensm-debuginfo? [y/N]:n
Install opensm-static? [y/N]:n
Install compat-dapl? [y/N]:n
Install compat-dapl-devel? [y/N]:n
Install dapl? [y/N]:n
Install dapl-devel? [y/N]:n
Install dapl-devel-static? [y/N]:n
Install dapl-utils? [y/N]:n
Install dapl-debuginfo? [y/N]:n
Install perftest? [y/N]:n
Install mstflint? [y/N]:y
Install tvflash? [y/N]:n
Install qlvnictools? [y/N]:n
Install sdpnetstat? [y/N]:n
Install srptools? [y/N]:n
Install rds-tools? [y/N]:n
Install rnfs-utils? [y/N]:y
Install ibutils? [y/N]:y
Install infiniband-diags? [y/N]:y
Install qperf? [y/N]:n
Install qperf-debuginfo? [y/N]:n
Install ofed-docs? [y/N]:y
Install ofed-scripts? [y/N]:y
Install tgt-generic? [y/N]:n
Install mpi-selector? [y/N]:n
Install openmpi_gcc? [y/N]:n
Install mpitests_openmpi_gcc? [y/N]:n
Please enter the OFED installation directory: [/usr]: