[nfs-rdma-devel] [ofa-general] Status of NFS-RDMA ?

[Pawel Dziekonski]

On Wed, 06 Feb 2008 at 10:17:24AM -0500, James Lentini wrote:

> > > One more question. Looking back at your NFS_START_SERVER script, I
> > > noticed that you are starting the NFS server "by hand" and not using
> > > your distro's scripts. I'm wondering if there if there is some
> > > aspect of the server configuration missing. Are you able to do a
> > > normal TCP or UDP mount of the server?
> > 
> > I'm doing this "by hand" because there is no nfs-utils 1.1.1 RPM 
> > package for my distro (Scientific Linux 4.5 - clone of RH 4u5), so I 
> > have uninstalled old nfs-utils RPM and compiled+installed new 
> > version by hand.
> 
> To be precise, you do not need nfs-utils 1.1.1 installed on your NFS 
> server. You only need the mount.nfs command (not event the whole 
> package) from nfs-util 1.1.1 or greater on your NFS client so you can 
> use the "string" mount API.

oh man, this is a good news.


> > regular nfs mount without rdma works like a charm:
> > 
> > # mount -v -t nfs 10.2.2.1:/scratch /mnt
> > mount.nfs: trying 10.2.2.1 prog 100003 vers 3 prot TCP port 2049
> > mount.nfs: trying 10.2.2.1 prog 100005 vers 3 prot UDP port 45151
> > 10.2.2.1:/scratch on /mnt type nfs (rw)
> > # ls -l /mnt
> > total 16
> > drwx------  2 root root 16384 Jan 25 16:29 lost+found/
> > # touch /mnt/qqq
> > # ls -l /mnt
> > total 16
> > drwx------  2 root root 16384 Jan 25 16:29 lost+found/
> > -rw-r--r--  1 root root     0 Feb  6 12:39 qqq
> > # umount /mnt
> 
> Good. If I remember correctly, 10.2.2.1, is the IPoIB interface on the 
> server. It looks like you are doing the mount over IPoIB. Given that 
> this works, it would appear that your export list is setup correctly.
> 
> By the way, do these machines have ethernet interfaces? Are the 
> Ethernet and IPoIB IPs on different subnets?

# ip a
1: lo: <LOOPBACK,UP,10000> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:30:48:7a:42:24 brd ff:ff:ff:ff:ff:ff
    inet 10.255.255.221/8 brd 10.255.255.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:30:48:7a:42:25 brd ff:ff:ff:ff:ff:ff
4: ib0: <BROADCAST,MULTICAST,UP,10000> mtu 2044 qdisc pfifo_fast qlen 128
    link/[32] 80:00:04:04:fe:80:00:00:00:00:00:00:00:30:48:7a:42:24:00:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 10.2.2.1/24 brd 10.2.2.255 scope global ib0

eth0 has to be in /8 network but I can change IPs on IB network if
there is such a need.

> The client's connection request is being refused. This looks like a 
> server problem.
> 
> On the server, what is the output of:
> 
>  cat /proc/fs/nfsd/portlist

# cat /proc/fs/nfsd/portlist
tcp 0.0.0.0, port=2049
udp 0.0.0.0, port=2049

# echo rdma 2050 > /proc/fs/nfsd/portlist
# cat /proc/fs/nfsd/portlist
tcp 0.0.0.0, port=2049
udp 0.0.0.0, port=2049

:o

>  cat /proc/sys/sunrpc/transports

# cat /proc/sys/sunrpc/transports
tcp 1048576
udp 32768
rdma 1048576

>  ps x | grep nfsd

# ps x | grep nfsd
 5155 ?        S      0:00 [nfsd]
 5727 pts/0    R+     0:00 grep nfsd

>  showmount -e 10.2.2.1

# showmount -e 10.2.2.1
Export list for 10.2.2.1:
/scratch 10.2.2.2,10.255.255.222

-- 
Pawel Dziekonski <pawel.dziekonski at wcss.pl>
Wroclaw Centre for Networking & Supercomputing, HPC Department
Politechnika Wr., pl. Grunwaldzki 9, bud. D2/101, 50-377 Wroclaw, POLAND
phone: +48 71 3202043, fax: +48 71 3225797, http://www.wcss.wroc.pl