[nfs-rdma-devel] [ofa-general] Status of NFS-RDMA ?

[James Lentini]

On Wed, 6 Feb 2008, Pawel Dziekonski wrote:

> On Tue, 05 Feb 2008 at 10:51:54AM -0500, James Lentini wrote:
> 
> > Looks like we'll need more data. Can you try the mount again, but this 
> > time with:
> > 
> >  echo 32767 > /proc/sys/sunrpc/rpc_debug
> >  echo 65535 > /proc/sys/sunrpc/nfs_debug
> > This should produce a lot of data.
> 
> > One more question. Looking back at your NFS_START_SERVER script, I
> > noticed that you are starting the NFS server "by hand" and not using
> > your distro's scripts. I'm wondering if there if there is some
> > aspect of the server configuration missing. Are you able to do a
> > normal TCP or UDP mount of the server?
> 
> I'm doing this "by hand" because there is no nfs-utils 1.1.1 RPM 
> package for my distro (Scientific Linux 4.5 - clone of RH 4u5), so I 
> have uninstalled old nfs-utils RPM and compiled+installed new 
> version by hand.

To be precise, you do not need nfs-utils 1.1.1 installed on your NFS 
server. You only need the mount.nfs command (not event the whole 
package) from nfs-util 1.1.1 or greater on your NFS client so you can 
use the "string" mount API.

Our instructions should note this. I'll update them.

> regular nfs mount without rdma works like a charm:
> 
> # mount -v -t nfs 10.2.2.1:/scratch /mnt
> mount.nfs: trying 10.2.2.1 prog 100003 vers 3 prot TCP port 2049
> mount.nfs: trying 10.2.2.1 prog 100005 vers 3 prot UDP port 45151
> 10.2.2.1:/scratch on /mnt type nfs (rw)
> # ls -l /mnt
> total 16
> drwx------  2 root root 16384 Jan 25 16:29 lost+found/
> # touch /mnt/qqq
> # ls -l /mnt
> total 16
> drwx------  2 root root 16384 Jan 25 16:29 lost+found/
> -rw-r--r--  1 root root     0 Feb  6 12:39 qqq
> # umount /mnt

Good. If I remember correctly, 10.2.2.1, is the IPoIB interface on the 
server. It looks like you are doing the mount over IPoIB. Given that 
this works, it would appear that your export list is setup correctly.

By the way, do these machines have ethernet interfaces? Are the 
Ethernet and IPoIB IPs on different subnets?

> with rdma:
> 
> # dmesg -c
> # echo 32767 > /proc/sys/sunrpc/rpc_debug
> # echo 65535 > /proc/sys/sunrpc/nfs_debug
> # mount.nfs 10.2.2.1:/scratch /mnt -i -o rdma,port=2050 -v
> mount.nfs: timeout set for Wed Feb  6 12:45:12 2008
> mount.nfs: text-based options: 'rdma,port=2050,addr=10.2.2.1'
> mount.nfs: internal error
> 
> dmesg log: https://cefeid.wcss.wroc.pl/d/tmp/dmesg.log

The client's connection request is being refused. This looks like a 
server problem.

On the server, what is the output of:

 cat /proc/fs/nfsd/portlist
 cat /proc/sys/sunrpc/transports
 ps x | grep nfsd
 showmount -e 10.2.2.1