<div> Dear Open Fabrics members.</div><div><br></div><div>I'm newbie about Infiniband and Open Fabrics.</div><div><br></div><div>I'm trying to setup NFS/RDMA on RHEL 5.5 x86_64</div><div><br></div><div><br></div><div>
I'm reading this document</div><div>/usr/share/doc/ofed-docs-1.5.2/nfs-rdma.release-notes.txt</div><div><br></div><div><br></div><div>1) My environment </div><div><br></div><div> RHEL-5.5 x86_64</div><div> OFED-1.5.2rc7</div>
<div><br></div><div> I executed the following command to install OFED</div><div><br></div><div> # cd OFED-1.5.2rc7</div><div> # ./<a href="http://install.pl">install.pl</a></div><div><br></div><div> OFED Distribution Software Installation Menu</div>
<div> </div><div> 1) Basic (OFED modules and basic user level libraries)</div><div> 2) HPC (OFED modules and libraries, MPI and diagnostic tools)</div><div> 3) All packages (all of Basic, HPC)</div>
<div> 4) Customize</div><div> </div><div> Q) Exit</div><div> </div><div> Select Option [1-4]:3</div><div> </div><div> Please choose an implementation of MVAPICH2:</div>
<div> </div><div> 1) OFA (IB and iWARP)</div><div> 2) uDAPL</div><div> Implementation [1]: </div><div> </div><div> Enable ROMIO support [Y/n]: </div><div> </div><div> Enable shared library support [Y/n]: </div>
<div> </div><div> Enable Checkpoint-Restart support [y/N]: </div><div> </div><div> Below is the list of OFED packages that you have chosen</div><div> (some may have been added by the installer due to package dependencies):</div>
<div><br></div><div> and reboot IPoIB work fine</div><div><br></div><div><br></div><div><br></div><div>2) Question</div><div><br></div><div> I setuped NFS/RMDA step-by-setp to read nfs-rdma.release-notes.txt</div><div>
But it does not work correctly. I can't know what is wrong.</div><div><br></div><div><br></div><div><br></div><div><br></div><div> 1) mount.rnfs proto option does not work nfs-rdma.release-notes.txt</div><div><br></div>
<div> mount.rnfs <IPoIB-server-name-or-address>:/<export> /mnt -o proto=rdma,port=20049</div><div><br></div><div> I executed mount.rnfs, the following error has occured. </div><div><br></div><div> mount.rnfs 192.168.100.231:/nfstest /mnt -o port=20049,proto=rdma</div>
<div> mount.rnfs: Bad nfs mount parameter: proto</div><div><br></div><div> # /sbin/mount.rnfs -V</div><div> mount.rnfs (linux nfs-utils 1.1.5)</div><div><br></div><div><br></div><div> 2) How to check rmda port ??</div>
<div><br></div><div> echo rdma 20049 > /proc/fs/nfsd/portlist</div><div><br></div><div> Could you tell me how to check it ( like netstat )</div><div><br></div><div> I can't find 20049 port </div><div> netstat -na | grep 20049</div>
<div><br></div><div><br></div><div>Thank you for your information .</div><div><br></div><div>Sincerely </div><div><br></div><div><br></div><div>--</div><div>Hiroyuki Sato</div><div><br></div><div><br></div><div><br></div>
<div><br></div><div><div># /sbin/lsmod | egrep '(ib_mthca|ib_ipoib|svcrdma)'</div><div>svcrdma 70208 1 </div><div>rdma_cm 73492 4 xprtrdma,svcrdma,rdma_ucm,ib_sdp</div><div>ib_ipoib 114016 0 </div>
<div>ib_cm 71592 2 rdma_cm,ib_ipoib</div><div>ib_sa 75784 4 rdma_ucm,rdma_cm,ib_ipoib,ib_cm</div><div>ib_mthca 157220 0 </div><div>ib_mad 72100 6 ib_cm,ib_sa,ib_umad,ib_qib,mlx4_ib,ib_mthca</div>
<div>ib_core 109440 17 xprtrdma,svcrdma,rdma_ucm,ib_sdp,rdma_cm,iw_cm,ib_ipoib,ib_cm,ib_sa,ib_uverbs,ib_umad,iw_nes,iw_cxgb3,ib_qib,mlx4_ib,ib_mthca,ib_mad</div><div>sunrpc 245672 17 xprtrdma,svcrdma,nfsd,nfs,nfs_acl,auth_rpcgss,lockd</div>
<div>ipoib_helper 35728 2 ib_ipoib</div><div>ipv6 435361 35 nfs,rdma_cm,ib_addr,ib_ipoib,lockd</div><div><br></div><div># cat /sys/class/infiniband/mthca0/ports/1/state </div><div>4: ACTIVE</div>
<div># /sbin/ifconfig ib0</div><div>ib0 Link encap:InfiniBand HWaddr 80:00:04:04:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 </div><div> inet addr:192.168.100.231 Bcast:192.168.100.255 Mask:255.255.255.0</div>
<div> inet6 addr: fe80::202:c902:22:eb9/64 Scope:Link</div><div> UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1</div><div> RX packets:45 errors:0 dropped:0 overruns:0 frame:0</div><div> TX packets:33 errors:0 dropped:6 overruns:0 carrier:0</div>
<div> collisions:0 txqueuelen:256 </div><div> RX bytes:6989 (6.8 KiB) TX bytes:2860 (2.7 KiB)</div><div><br></div><div># /sbin/service nfs status</div><div>rpc.mountd (pid 5280) is running...</div><div>
nfsd (pid 5277 5276 5275 5274 5273 5272 5271 5270) is running...</div><div>rpc.rquotad (pid 5205) is running...</div><div><br></div><div><br></div><div># cat /proc/fs/nfsd/portlist</div><div>rdma 20049</div><div>tcp 2049</div>
<div>udp 2049</div><div><br></div><div># netstat -na -p | grep 20049</div><div>no output</div><div><br></div><div><br></div><div># netstat -na -p | grep 2049 </div><div><br></div><div>tcp 0 0 <a href="http://0.0.0.0:2049">0.0.0.0:2049</a> 0.0.0.0:* LISTEN - </div>
<div>udp 0 0 <a href="http://0.0.0.0:2049">0.0.0.0:2049</a> 0.0.0.0:* - </div></div><div><br></div><div><br></div><div><br></div><div><br></div><div>
<br></div><div>################################################################################</div><div># #</div><div># NFS/RDMA README #</div>
<div># #</div><div>################################################################################</div><div><br></div><div> Author: NetApp and Open Grid Computing</div>
<div> </div><div> Adapted for OFED 1.5.1 (from linux-2.6.30/Documentation/filesystems/nfs-rdma.txt)</div><div> by Jon Mason</div><div><br></div><div>Table of Contents</div><div>~~~~~~~~~~~~~~~~~</div><div> - Overview</div>
<div> - OFED 1.5.1 limitations</div><div> - Getting Help</div><div> - Installation</div><div> - Check RDMA and NFS Setup</div><div> - NFS/RDMA Setup</div><div><br></div><div>Overview</div><div>~~~~~~~~</div><div><br></div>
<div> This document describes how to install and setup the Linux NFS/RDMA client</div><div> and server software.</div><div><br></div><div> The NFS/RDMA client was first included in Linux 2.6.24. The NFS/RDMA server</div>
<div> was first included in the following release, Linux 2.6.25.</div><div><br></div><div> In our testing, we have obtained excellent performance results (full 10Gbit</div><div> wire bandwidth at minimal client CPU) under many workloads. The code passes</div>
<div> the full Connectathon test suite and operates over both Infiniband and iWARP</div><div> RDMA adapters.</div><div><br></div><div>OFED 1.5.1 limitations:</div><div>~~~~~~~~~~~~~~~~~~~~~</div><div> NFS-RDMA is supported for the following releases:</div>
<div> - Redhat Enterprise Linux (RHEL) version 5.2</div><div> - Redhat Enterprise Linux (RHEL) version 5.3</div><div> - Redhat Enterprise Linux (RHEL) version 5.4</div><div> - SUSE Linux Enterprise Server (SLES) version 11</div>
<div><br></div><div> And the following <a href="http://kernel.org">kernel.org</a> kernels:</div><div> - 2.6.22</div><div> - 2.6.25</div><div> - 2.6.30</div><div><br></div><div> All other Linux Distrubutions and kernel versions are NOT supported on OFED</div>
<div> 1.5.1</div><div><br></div><div>Getting Help</div><div>~~~~~~~~~~~~</div><div><br></div><div> If you get stuck, you can ask questions on the</div><div> <a href="mailto:nfs-rdma-devel@lists.sourceforge.net">nfs-rdma-devel@lists.sourceforge.net</a>, or <a href="mailto:linux-rdma@vger.kernel.org">linux-rdma@vger.kernel.org</a></div>
<div> mailing lists.</div><div><br></div><div>Installation</div><div>~~~~~~~~~~~~</div><div><br></div><div> These instructions are a step by step guide to building a machine for</div><div> use with NFS/RDMA.</div><div>
<br></div><div> - Install an RDMA device</div><div><br></div><div> Any device supported by the drivers in drivers/infiniband/hw is acceptable.</div><div><br></div><div> Testing has been performed using several Mellanox-based IB cards and </div>
<div> the Chelsio cxgb3 iWARP adapter.</div><div><br></div><div> - Install OFED 1.5.1</div><div><br></div><div> NFS/RDMA has been tested on RHEL5.2, RHEL 5.3, RHEL5.4, SLES11,</div><div> kernels 2.6.22, 2.6.25, and 2.6.30. On these kernels,</div>
<div> NFS-RDMA will be installed by default if you simply select "install all",</div><div> and can be specifically included by a "custom" install.</div><div><br></div><div> In addition, the install script will install a version of the nfs-utils that</div>
<div> is required for NFS/RDMA. The binary installed will be named "mount.rnfs".</div><div> This version is not necessary for Linux Distributions with nfs-utils 1.1 or</div><div> later.</div><div><br></div>
<div> Upon successful installation, the nfs kernel modules will be placed in the</div><div> directory /lib/modules/'uname -a'/updates. It is recommended that you reboot</div><div> to ensure that the correct modules are loaded.</div>
<div><br></div><div>Check RDMA and NFS Setup</div><div>~~~~~~~~~~~~~~~~~~~~~~~~</div><div><br></div><div> Before configuring the NFS/RDMA software, it is a good idea to test</div><div> your new kernel to ensure that the kernel is working correctly.</div>
<div> In particular, it is a good idea to verify that the RDMA stack</div><div> is functioning as expected and standard NFS over TCP/IP and/or UDP/IP</div><div> is working properly.</div><div><br></div><div> - Check RDMA Setup</div>
<div><br></div><div> If you built the RDMA components as modules, load them at</div><div> this time. For example, if you are using a Mellanox Tavor/Sinai/Arbel</div><div> card:</div><div><br></div><div> $ modprobe ib_mthca</div>
<div> $ modprobe ib_ipoib</div><div><br></div><div> If you are using InfiniBand, make sure there is a Subnet Manager (SM)</div><div> running on the network. If your IB switch has an embedded SM, you can</div><div>
use it. Otherwise, you will need to run an SM, such as OpenSM, on one</div><div> of your end nodes.</div><div><br></div><div> If an SM is running on your network, you should see the following:</div><div><br></div>
<div> $ cat /sys/class/infiniband/driverX/ports/1/state</div><div> 4: ACTIVE</div><div><br></div><div> where driverX is mthca0, ipath5, ehca3, etc.</div><div><br></div><div> To further test the InfiniBand software stack, use IPoIB (this</div>
<div> assumes you have two IB hosts named host1 and host2):</div><div><br></div><div> host1$ ifconfig ib0 a.b.c.x</div><div> host2$ ifconfig ib0 a.b.c.y</div><div> host1$ ping a.b.c.y</div><div> host2$ ping a.b.c.x</div>
<div><br></div><div> For other device types, follow the appropriate procedures.</div><div><br></div><div> - Check NFS Setup</div><div><br></div><div> For the NFS components enabled above (client and/or server),</div>
<div> test their functionality over standard Ethernet using TCP/IP or UDP/IP.</div><div><br></div><div>NFS/RDMA Setup</div><div>~~~~~~~~~~~~~~</div><div><br></div><div> We recommend that you use two machines, one to act as the client and</div>
<div> one to act as the server.</div><div><br></div><div> One time configuration:</div><div><br></div><div> - On the server system, configure the /etc/exports file and</div><div> start the NFS/RDMA server.</div><div>
<br></div><div> Exports entries with the following formats have been tested:</div><div><br></div><div> /vol0 192.168.0.47(fsid=0,rw,async,insecure,no_root_squash)</div><div> /vol0 <a href="http://192.168.0.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash)">192.168.0.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash)</a></div>
<div><br></div><div> The IP address(es) is(are) the client's IPoIB address for an InfiniBand</div><div> HCA or the client's iWARP address(es) for an RNIC.</div><div><br></div><div> NOTE: The "insecure" option must be used because the NFS/RDMA client does</div>
<div> not use a reserved port.</div><div><br></div><div> Each time a machine boots:</div><div><br></div><div> - Load and configure the RDMA drivers</div><div><br></div><div> For InfiniBand using a Mellanox adapter:</div>
<div><br></div><div> $ modprobe ib_mthca</div><div> $ modprobe ib_ipoib</div><div> $ ifconfig ib0 a.b.c.d</div><div><br></div><div> NOTE: use unique addresses for the client and server</div><div><br></div><div>
- Start the NFS server</div><div><br></div><div> Load the RDMA transport module:</div><div><br></div><div> $ modprobe svcrdma</div><div><br></div><div> Start the server:</div><div><br></div><div> $ /etc/init.d/nfsserver start</div>
<div><br></div><div> or</div><div><br></div><div> $ service nfs start</div><div><br></div><div> Instruct the server to listen on the RDMA transport:</div><div><br></div><div> $ echo rdma 20049 > /proc/fs/nfsd/portlist</div>
<div><br></div><div> NOTE for SLES10 servers: The nfs start scripts on most distro's start</div><div> rpc.statd by default. However, the in-kernel lockd that was in SLES10 has</div><div> been removed in the new kernels. Since OFED is back-porting the new code to</div>
<div> the older distro's, there is no in-kernel lockd in SLES10 and the SLES10</div><div> nfsserver scripts do not know the need to start it. Therefore, the</div><div> nfsserver scripts will be modified when the rnfs-utils rpm is installed to</div>
<div> start/stop rpc.statd.</div><div><br></div><div> - On the client system</div><div><br></div><div> Load the RDMA client module:</div><div><br></div><div> $ modprobe xprtrdma</div><div><br></div><div> Mount the NFS/RDMA server:</div>
<div><br></div><div> $ mount.rnfs <IPoIB-server-name-or-address>:/<export> /mnt -o proto=rdma,port=20049 </div><div><br></div><div> NOTE: For kernels < 2.6.23, the "-i" flag must be passed into mount.rnfs.</div>
<div> This option allows the mount command to ignore the kernel version check. If</div><div> not disabled, the check will prevent passing arguments to the kernel and not</div><div> allow the updated version of NFS to accept the "rdma" NFS option.</div>
<div><br></div><div> To verify that the mount is using RDMA, run "cat /proc/mounts" and check</div><div> the "proto" field for the given mount.</div><div><br></div><div> Congratulations! You're using NFS/RDMA!</div>
<div><br></div><div>Known Issues</div><div>~~~~~~~~~~~~~~~~~~~~~~~~</div><div><br></div><div>If you're running NFSRDMA over Chelsio's T3 RNIC and your cients are using</div><div>a 64KB page size (like PPC64 and IA64 systems) and your server is using a</div>
<div>4KB page size (like i386 and X86_64), then you need to mount the server</div><div>using rsize=32768,wsize=32768 to avoid overrunning the Chelsio RNIC fast</div><div>register limits. This is a known firmware limitation in the Chelsio RNIC.</div>
<div><br></div><div>Running NFSRDMA over Mellanox's ConnectX HCA requires that the adapter firmware</div><div>be 2.7.0 or greater on all NFS clients and servers. Firmware 2.6.0 has known</div><div>issues that prevent the RDMA connection from being established. Firmware 2.7.0</div>
<div>has resolved these issues.</div><div><br></div><div>IPv6 support requires portmap that supports version 4. Portmap included in RHEL5</div><div>and SLES10 only supports version 2. Without version 4 support, the following</div>
<div>error will be logged:</div><div> svc: failed to register lockdv1 (errno 97).</div><div>This error will not affect IPv4 support.</div>