[ewg] [Q] NFS/RDMA on RHEL 5.5 x86_64

Hiroyuki Sato hiroysato at gmail.com
Sat Sep 18 03:12:06 PDT 2010


 Dear Open Fabrics members.

I'm newbie about Infiniband and Open Fabrics.

I'm trying to setup NFS/RDMA on RHEL 5.5 x86_64


I'm reading this document
/usr/share/doc/ofed-docs-1.5.2/nfs-rdma.release-notes.txt


1) My environment

  RHEL-5.5 x86_64
  OFED-1.5.2rc7

    I executed the following command to install OFED

      # cd OFED-1.5.2rc7
      # ./install.pl

        OFED Distribution Software Installation Menu

           1) Basic (OFED modules and basic user level libraries)
           2) HPC (OFED modules and libraries, MPI and diagnostic tools)
           3) All packages (all of Basic, HPC)
           4) Customize

           Q) Exit

        Select Option [1-4]:3

        Please choose an implementation of MVAPICH2:

        1) OFA (IB and iWARP)
        2) uDAPL
        Implementation [1]:

        Enable ROMIO support [Y/n]:

        Enable shared library support [Y/n]:

        Enable Checkpoint-Restart support [y/N]:

        Below is the list of OFED packages that you have chosen
        (some may have been added by the installer due to package
dependencies):

    and reboot IPoIB work fine



2) Question

  I setuped NFS/RMDA step-by-setp to read nfs-rdma.release-notes.txt
  But it does not work correctly. I can't know what is wrong.




  1) mount.rnfs proto option does not work nfs-rdma.release-notes.txt

    mount.rnfs <IPoIB-server-name-or-address>:/<export> /mnt -o
proto=rdma,port=20049

    I executed mount.rnfs, the following error has occured.

      mount.rnfs 192.168.100.231:/nfstest /mnt  -o port=20049,proto=rdma
      mount.rnfs: Bad nfs mount parameter: proto

      # /sbin/mount.rnfs -V
      mount.rnfs (linux nfs-utils 1.1.5)


  2) How to check rmda port ??

    echo rdma 20049 > /proc/fs/nfsd/portlist

    Could you tell me how to check it ( like netstat )

    I can't find 20049 port
      netstat -na | grep 20049


Thank you for your information .

Sincerely


--
Hiroyuki Sato




# /sbin/lsmod | egrep '(ib_mthca|ib_ipoib|svcrdma)'
svcrdma                70208  1
rdma_cm                73492  4 xprtrdma,svcrdma,rdma_ucm,ib_sdp
ib_ipoib              114016  0
ib_cm                  71592  2 rdma_cm,ib_ipoib
ib_sa                  75784  4 rdma_ucm,rdma_cm,ib_ipoib,ib_cm
ib_mthca              157220  0
ib_mad                 72100  6 ib_cm,ib_sa,ib_umad,ib_qib,mlx4_ib,ib_mthca
ib_core               109440  17
xprtrdma,svcrdma,rdma_ucm,ib_sdp,rdma_cm,iw_cm,ib_ipoib,ib_cm,ib_sa,ib_uverbs,ib_umad,iw_nes,iw_cxgb3,ib_qib,mlx4_ib,ib_mthca,ib_mad
sunrpc                245672  17
xprtrdma,svcrdma,nfsd,nfs,nfs_acl,auth_rpcgss,lockd
ipoib_helper           35728  2 ib_ipoib
ipv6                  435361  35 nfs,rdma_cm,ib_addr,ib_ipoib,lockd

# cat /sys/class/infiniband/mthca0/ports/1/state
4: ACTIVE
# /sbin/ifconfig ib0
ib0       Link encap:InfiniBand  HWaddr
80:00:04:04:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
          inet addr:192.168.100.231  Bcast:192.168.100.255
 Mask:255.255.255.0
          inet6 addr: fe80::202:c902:22:eb9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:65520  Metric:1
          RX packets:45 errors:0 dropped:0 overruns:0 frame:0
          TX packets:33 errors:0 dropped:6 overruns:0 carrier:0
          collisions:0 txqueuelen:256
          RX bytes:6989 (6.8 KiB)  TX bytes:2860 (2.7 KiB)

# /sbin/service nfs status
rpc.mountd (pid 5280) is running...
nfsd (pid 5277 5276 5275 5274 5273 5272 5271 5270) is running...
rpc.rquotad (pid 5205) is running...


# cat /proc/fs/nfsd/portlist
rdma 20049
tcp 2049
udp 2049

# netstat -na -p | grep 20049
no output


# netstat -na -p | grep 2049

tcp        0      0 0.0.0.0:2049                0.0.0.0:*
LISTEN      -
udp        0      0 0.0.0.0:2049                0.0.0.0:*
            -





################################################################################
#
   #
#                               NFS/RDMA README
   #
#
   #
################################################################################

 Author: NetApp and Open Grid Computing

 Adapted for OFED 1.5.1 (from
linux-2.6.30/Documentation/filesystems/nfs-rdma.txt)
 by Jon Mason

Table of Contents
~~~~~~~~~~~~~~~~~
 - Overview
 - OFED 1.5.1 limitations
 - Getting Help
 - Installation
 - Check RDMA and NFS Setup
 - NFS/RDMA Setup

Overview
~~~~~~~~

  This document describes how to install and setup the Linux NFS/RDMA client
  and server software.

  The NFS/RDMA client was first included in Linux 2.6.24. The NFS/RDMA
server
  was first included in the following release, Linux 2.6.25.

  In our testing, we have obtained excellent performance results (full
10Gbit
  wire bandwidth at minimal client CPU) under many workloads. The code
passes
  the full Connectathon test suite and operates over both Infiniband and
iWARP
  RDMA adapters.

OFED 1.5.1 limitations:
~~~~~~~~~~~~~~~~~~~~~
  NFS-RDMA is supported for the following releases:
  - Redhat Enterprise Linux (RHEL) version 5.2
  - Redhat Enterprise Linux (RHEL) version 5.3
  - Redhat Enterprise Linux (RHEL) version 5.4
  - SUSE Linux Enterprise Server (SLES) version 11

  And the following kernel.org kernels:
  - 2.6.22
  - 2.6.25
  - 2.6.30

  All other Linux Distrubutions and kernel versions are NOT supported on
OFED
  1.5.1

Getting Help
~~~~~~~~~~~~

  If you get stuck, you can ask questions on the
  nfs-rdma-devel at lists.sourceforge.net, or linux-rdma at vger.kernel.org
  mailing lists.

Installation
~~~~~~~~~~~~

  These instructions are a step by step guide to building a machine for
  use with NFS/RDMA.

  - Install an RDMA device

    Any device supported by the drivers in drivers/infiniband/hw is
acceptable.

    Testing has been performed using several Mellanox-based IB cards and
    the Chelsio cxgb3 iWARP adapter.

  - Install OFED 1.5.1

    NFS/RDMA has been tested on RHEL5.2, RHEL 5.3, RHEL5.4, SLES11,
    kernels 2.6.22, 2.6.25, and 2.6.30.  On these kernels,
    NFS-RDMA will be installed by default if you simply select "install
all",
    and can be specifically included by a "custom" install.

    In addition, the install script will install a version of the nfs-utils
that
    is required for NFS/RDMA.  The binary installed will be named
"mount.rnfs".
    This version is not necessary for Linux Distributions with nfs-utils 1.1
or
    later.

    Upon successful installation, the nfs kernel modules will be placed in
the
    directory /lib/modules/'uname -a'/updates. It is recommended that you
reboot
    to ensure that the correct modules are loaded.

Check RDMA and NFS Setup
~~~~~~~~~~~~~~~~~~~~~~~~

    Before configuring the NFS/RDMA software, it is a good idea to test
    your new kernel to ensure that the kernel is working correctly.
    In particular, it is a good idea to verify that the RDMA stack
    is functioning as expected and standard NFS over TCP/IP and/or UDP/IP
    is working properly.

  - Check RDMA Setup

    If you built the RDMA components as modules, load them at
    this time. For example, if you are using a Mellanox Tavor/Sinai/Arbel
    card:

    $ modprobe ib_mthca
    $ modprobe ib_ipoib

    If you are using InfiniBand, make sure there is a Subnet Manager (SM)
    running on the network. If your IB switch has an embedded SM, you can
    use it. Otherwise, you will need to run an SM, such as OpenSM, on one
    of your end nodes.

    If an SM is running on your network, you should see the following:

    $ cat /sys/class/infiniband/driverX/ports/1/state
    4: ACTIVE

    where driverX is mthca0, ipath5, ehca3, etc.

    To further test the InfiniBand software stack, use IPoIB (this
    assumes you have two IB hosts named host1 and host2):

    host1$ ifconfig ib0 a.b.c.x
    host2$ ifconfig ib0 a.b.c.y
    host1$ ping a.b.c.y
    host2$ ping a.b.c.x

    For other device types, follow the appropriate procedures.

  - Check NFS Setup

    For the NFS components enabled above (client and/or server),
    test their functionality over standard Ethernet using TCP/IP or UDP/IP.

NFS/RDMA Setup
~~~~~~~~~~~~~~

  We recommend that you use two machines, one to act as the client and
  one to act as the server.

  One time configuration:

  - On the server system, configure the /etc/exports file and
    start the NFS/RDMA server.

    Exports entries with the following formats have been tested:

    /vol0   192.168.0.47(fsid=0,rw,async,insecure,no_root_squash)
    /vol0
192.168.0.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash)

    The IP address(es) is(are) the client's IPoIB address for an InfiniBand
    HCA or the client's iWARP address(es) for an RNIC.

    NOTE: The "insecure" option must be used because the NFS/RDMA client
does
    not use a reserved port.

 Each time a machine boots:

  - Load and configure the RDMA drivers

    For InfiniBand using a Mellanox adapter:

    $ modprobe ib_mthca
    $ modprobe ib_ipoib
    $ ifconfig ib0 a.b.c.d

    NOTE: use unique addresses for the client and server

  - Start the NFS server

    Load the RDMA transport module:

    $ modprobe svcrdma

    Start the server:

    $ /etc/init.d/nfsserver start

    or

    $ service nfs start

    Instruct the server to listen on the RDMA transport:

    $ echo rdma 20049 > /proc/fs/nfsd/portlist

    NOTE for SLES10 servers: The nfs start scripts on most distro's start
    rpc.statd by default.  However, the in-kernel lockd that was in SLES10
has
    been removed in the new kernels.  Since OFED is back-porting the new
code to
    the older distro's, there is no in-kernel lockd in SLES10 and the SLES10
    nfsserver scripts do not know the need to start it.  Therefore, the
    nfsserver scripts will be modified when the rnfs-utils rpm is installed
to
    start/stop rpc.statd.

  - On the client system

    Load the RDMA client module:

    $ modprobe xprtrdma

    Mount the NFS/RDMA server:

    $ mount.rnfs <IPoIB-server-name-or-address>:/<export> /mnt -o
proto=rdma,port=20049

    NOTE: For kernels < 2.6.23, the "-i" flag must be passed into
mount.rnfs.
    This option allows the mount command to ignore the kernel version check.
 If
    not disabled, the check will prevent passing arguments to the kernel and
not
    allow the updated version of NFS to accept the "rdma" NFS option.

    To verify that the mount is using RDMA, run "cat /proc/mounts" and check
    the "proto" field for the given mount.

  Congratulations! You're using NFS/RDMA!

Known Issues
~~~~~~~~~~~~~~~~~~~~~~~~

If you're running NFSRDMA over Chelsio's T3 RNIC and your cients are using
a 64KB page size (like PPC64 and IA64 systems) and your server is using a
4KB page size (like i386 and X86_64), then you need to mount the server
using rsize=32768,wsize=32768 to avoid overrunning the Chelsio RNIC fast
register limits.  This is a known firmware limitation in the Chelsio RNIC.

Running NFSRDMA over Mellanox's ConnectX HCA requires that the adapter
firmware
be 2.7.0 or greater on all NFS clients and servers.  Firmware 2.6.0 has
known
issues that prevent the RDMA connection from being established.  Firmware
2.7.0
has resolved these issues.

IPv6 support requires portmap that supports version 4. Portmap included in
RHEL5
and SLES10 only supports version 2.  Without version 4 support, the
following
error will be logged:
       svc: failed to register lockdv1 (errno 97).
This error will not affect IPv4 support.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ewg/attachments/20100918/43d1f0fd/attachment.html>


More information about the ewg mailing list