[nfs-rdma-devel] [ofa-general] Status of NFS-RDMA ?

[James Lentini]

On Wed, 6 Feb 2008, Pawel Dziekonski wrote:

> On Wed, 06 Feb 2008 at 10:17:24AM -0500, James Lentini wrote:
> 
> > By the way, do these machines have ethernet interfaces? Are the 
> > Ethernet and IPoIB IPs on different subnets?
> 
> # ip a
> 1: lo: <LOOPBACK,UP,10000> mtu 16436 qdisc noqueue
>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>     inet 127.0.0.1/8 scope host lo
> 2: eth0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000
>     link/ether 00:30:48:7a:42:24 brd ff:ff:ff:ff:ff:ff
>     inet 10.255.255.221/8 brd 10.255.255.255 scope global eth0
> 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast qlen 1000
>     link/ether 00:30:48:7a:42:25 brd ff:ff:ff:ff:ff:ff
> 4: ib0: <BROADCAST,MULTICAST,UP,10000> mtu 2044 qdisc pfifo_fast qlen 128
>     link/[32] 80:00:04:04:fe:80:00:00:00:00:00:00:00:30:48:7a:42:24:00:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
>     inet 10.2.2.1/24 brd 10.2.2.255 scope global ib0
> 
> eth0 has to be in /8 network but I can change IPs on IB network if
> there is such a need.

The subneting scheme above looks like a potential problem. It looks 
like all packets to 10.x.y.z will be routed through the eth0 
interface. I'd suggest moving the IPoIB interface to a different 
private network, say 192.168.0.0.

> > The client's connection request is being refused. This looks like a 
> > server problem.
> > 
> > On the server, what is the output of:
> > 
> >  cat /proc/fs/nfsd/portlist
> 
> # cat /proc/fs/nfsd/portlist
> tcp 0.0.0.0, port=2049
> udp 0.0.0.0, port=2049
> 
> # echo rdma 2050 > /proc/fs/nfsd/portlist
> # cat /proc/fs/nfsd/portlist
> tcp 0.0.0.0, port=2049
> udp 0.0.0.0, port=2049
> 
> :o

You're right. This is a problem. You should have just received some 
instructions from Tom Tucker on how to fix this.

> >  cat /proc/sys/sunrpc/transports
> 
> # cat /proc/sys/sunrpc/transports
> tcp 1048576
> udp 32768
> rdma 1048576
> 
> >  ps x | grep nfsd
> 
> # ps x | grep nfsd
>  5155 ?        S      0:00 [nfsd]
>  5727 pts/0    R+     0:00 grep nfsd

This should work, but you will want more nfsd threads for performance.

I'd recommend going back to using the nfs server startup scripts from 
your distro. That should take care of this and any other setup details 
for you.