[nfs-rdma-devel] [ofa-general] Status of NFS-RDMA ?

[James Lentini]

On Tue, 5 Feb 2008, Pawel Dziekonski wrote:

> On Mon, 04 Feb 2008 at 05:39:25PM -0500, James Lentini wrote:
> 
> > I don't use OFED, but I'll share with you what I know. OFED 1.3 is 
> > based on 2.6.24, which is the same kernel Tom Tucker's git tree is 
> > based on. I would assume the two are compatible, but I have not tried 
> > it.
> > 
> > Do you have NFS/RDMA working at this point?
> 
> no - internal error again. :(
> 
> I have uninstalled OFED completely and I'm running pure Tom Tucker's
> kernel.
> 
> yum install opensm
> /etc/init.d/opensmd start
> cat /sys/class/infiniband/mthca0/ports/1/state
> 4: ACTIVE
> 
> ifconfig ib0
> ib0       Link encap:UNSPEC  HWaddr 80-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
>           inet addr:10.2.2.1  Bcast:10.2.2.255  Mask:255.255.255.0
>           UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
>           RX packets:8 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:128
>           RX bytes:784 (784.0 b)  TX bytes:552 (552.0 b)

Is this ib0 interface on the server?

Can you ping the server's IPoIB interface from your client?

Based on the output below, I think to these questions is yes.

> rpm -e nfs-utils
> cd nfs-utils-1.1.1/
> ./configure --prefix=/usr/local --disable-gss --disable-nfs4
> make
> make install
> cd ..
> 
> cat /etc/exports
> /scratch 10.255.255.222(no_subtree_check,rw,async) 10.2.2.2(no_subtree_check,rw,async,insecure,no_root_squash)
> 
> cat NFS_START_SERVER
> #!/bin/sh
> export PATH="/usr/local/sbin:/usr/local/bin:$PATH"
> /etc/rc.d/init.d/portmap restart
> modprobe svcrdma
> modprobe nfs
> umount /proc/fs/nfsd >&/dev/null
> mount -t nfsd nfsd /proc/fs/nfsd >&/dev/null
> mount |grep /proc/fs/nfsd
> exportfs -av
> rpc.mountd
> rpc.statd --no-notify
> rpc.nfsd
> sm-notify
> echo rdma 2050 > /proc/fs/nfsd/portlist
> 
> ./NFS_START_SERVER
> 
> same done on client.

What is the same on the client?

> cat NFS_START_CLIENT
> #!/bin/sh
> export PATH="/usr/local/sbin:/usr/local/bin:$PATH"
> /etc/rc.d/init.d/portmap restart
> modprobe svcrdma

You don't need to load the svcrdma module on the client. The svcrdma 
module is only needed on the server.

The client needs to have the xprtrdma code loaded. You appear to have 
built nfs as a module, so the xprtrdma code would also have been built 
as a module. Instead of loading scvrdma, you should run this here:

 modprobe xprtrdma

Did you see this message on the client? 

 RPC: transport (256) not supported

> modprobe nfs
> sm-notify
> 
> ./NFS_START_CLIENT
> 
> mount.nfs 10.2.2.1:/scratch /mnt -i -o rdma,port=2050 -v     
> mount.nfs: timeout set for Tue Feb  5 14:50:47 2008
> mount.nfs: text-based options: 'rdma,port=2050,addr=10.2.2.1'
> mount.nfs: internal error
> 
> mount("10.2.2.1:/scratch", "/mnt", "nfs", 0, "rdma,port=2050,addr=10.2.2.1") = -1 EIO (Input/output error)
> write(2, "mount.nfs: internal error\n", 26mount.nfs: internal error
> ) = 26
> exit_group(32)                          = ?
> Process 13170 detached
> 
> server logs:
> Feb  5 14:45:46 ib1 mountd[19614]: authenticated mount request from 10.2.2.2:988 for /scratch (/scratch)

If loading the xprtrdma module doesn't solve your problem, send the 
output from these commands:

 dmesc -c > /dev/null
 echo 1024 > /proc/sys/sunrpc/nfs_debug
 mount.nfs 10.2.2.1:/scratch /mnt -i -o rdma,port=2050 -v
 dmesg > output


> :(
> 
> -- 
> Pawel Dziekonski <pawel.dziekonski at wcss.pl>
> Wroclaw Centre for Networking & Supercomputing, HPC Department
> Politechnika Wr., pl. Grunwaldzki 9, bud. D2/101, 50-377 Wroclaw, POLAND
> phone: +48 71 3202043, fax: +48 71 3225797, http://www.wcss.wroc.pl
>