[ewg] [Fwd: nfs-rdma hanging with Ubuntu 9.10]
Ross Smith
myxiplx at googlemail.com
Tue Jan 26 08:29:26 PST 2010
No problem, reporting odd stuff is one of the few things I can contribute :)
$ uname -a
Linux ubuntu-server 2.6.31-14-generic #48-Ubuntu SMP Fri Oct 16
14:04:26 UTC 2009 i686 GNU/Linux
On Tue, Jan 26, 2010 at 4:26 PM, Tom Tucker <tom at opengridcomputing.com> wrote:
> Ross Smith wrote:
>>
>> Interesting, the single '>' didn't work for me, it removed the tcp and
>> udp entries, leaving me with just rdma. It looks like you do need the
>>
>
> Huh. Can you do a 'uname -a' for me. Someone has changed that. Thank you for
> the heads up.
>
> Tom
>
>>>> on Ubuntu 9.10.
>>>>
>>
>>
>> On Tue, Jan 26, 2010 at 4:20 PM, Tom Tucker <tom at opengridcomputing.com>
>> wrote:
>>
>>>
>>> Ross Smith wrote:
>>>
>>>>
>>>> No problem, but I think you need an extra > too :)
>>>>
>>>> # echo "rdma 20049" >> /proc/fs/nfsd/portlist
>>>>
>>>>
>>>
>>> Actually, it's not a "real" file. So just the single '>' will work fine.
>>> There is logic inside the kernel
>>> that handles the write and converts the 'rdma 20049' to calls inside the
>>> kernel that create a
>>> listening endpoint for the rdma transport.
>>>
>>>>
>>>> And that was enough to get me going, although there was one more step
>>>> I'd missed:
>>>>
>>>> # mount 192.168.101.5:/home/ross/nfsexport ./nfstest -o
>>>> proto=rdma,port=20049
>>>> mount.nfs: Operation not permitted
>>>>
>>>> Googling that lead me to modify /etc/exports on the server to add the
>>>> insecure option. With that added it works fine.
>>>>
>>>>
>>>>
>>>
>>> Awesome.
>>>
>>>
>>>>
>>>> Now I just need to get it connecting to Solaris without crashing the
>>>> server :)
>>>>
>>>> Many thanks for all the help.
>>>>
>>>> Ross
>>>>
>>>>
>>>>
>>>> On Tue, Jan 26, 2010 at 3:49 PM, Tom Tucker <tom at opengridcomputing.com>
>>>> wrote:
>>>>
>>>>
>>>>>
>>>>> Ross Smith wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> Hmm, the portlist doesn't look good:
>>>>>>
>>>>>> $ cat /proc/fs/nfsd/portlist
>>>>>> tcp 2049
>>>>>> udp 2049
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> No it looks great, that's an easy one! No one is listening on 20049, so
>>>>> you
>>>>> get 111 (ECONNREFUSED)
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> But attempting to modify that fails:
>>>>>>
>>>>>> # echo 20049 > /proc/fs/nfsd/portlist
>>>>>> -bash: echo: write error: Bad file descriptor
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> That's because I gave you the wrong syntax for the write command. It
>>>>> should
>>>>> be the following:
>>>>>
>>>>> # echo "rdma 20049" > /proc/fs/nfsd/portlist
>>>>>
>>>>> Sorry about that.
>>>>>
>>>>> Tom
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> And I get similar problems attempting to enable the debugging logs:
>>>>>>
>>>>>> # echo 32767 > /proc/sys/sunrpc/rpc_debug
>>>>>> -bash: /proc/sys/sunrpc/rpc_debug: Permission denied
>>>>>>
>>>>>> Up to that point through everything looks like it's loading fine:
>>>>>>
>>>>>> Ubuntu server:
>>>>>> ===========
>>>>>> # modprobe mlx4_ib
>>>>>> # modprobe ib_ipoib
>>>>>> # ifconfig ib0 192.168.101.5 netmask 255.255.255.0 up
>>>>>>
>>>>>> dmesg results:
>>>>>> [ 456.793661] mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0
>>>>>> (April
>>>>>> 4,
>>>>>> 2008)
>>>>>> [ 456.987043] ADDRCONF(NETDEV_UP): ib0: link is not ready
>>>>>> [ 459.988683] ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
>>>>>> [ 470.686631] ib0: no IPv6 routers present
>>>>>>
>>>>>> # modprobe svcrdma
>>>>>> # /etc/init.d/nfs-kernel-server restart
>>>>>>
>>>>>> dmesg:
>>>>>> [ 524.520198] nfsd: last server has exited, flushing export cache
>>>>>> [ 529.292366] svc: failed to register lockdv1 RPC service (errno 97).
>>>>>> [ 529.293289] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state
>>>>>> recovery directory
>>>>>> [ 529.293304] NFSD: starting 90-second grace period
>>>>>>
>>>>>> Ubuntu client:
>>>>>> ==========
>>>>>> # modprobe mlx4_ib
>>>>>> # modprobe ib_ipoib
>>>>>> # ifconfig ib0 192.168.101.4 netmask 255.255.255.0 up
>>>>>>
>>>>>> dmesg:
>>>>>> [ 97.576507] mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0
>>>>>> (April
>>>>>> 4,
>>>>>> 2008)
>>>>>> [ 97.769582] ADDRCONF(NETDEV_UP): ib0: link is not ready
>>>>>> [ 100.765318] ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
>>>>>> [ 110.899591] ib0: no IPv6 routers present
>>>>>>
>>>>>> # modprobe xprtrdma
>>>>>>
>>>>>> dmesg:
>>>>>> [ 169.269689] RPC: Registered udp transport module.
>>>>>> [ 169.269691] RPC: Registered tcp transport module.
>>>>>> [ 169.289755] RPC: Registered rdma transport module.
>>>>>>
>>>>>> Ross
>>>>>>
>>>>>>
>>>>>> On Tue, Jan 26, 2010 at 2:32 PM, Tom Tucker
>>>>>> <tom at opengridcomputing.com>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Ross Smith wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> A quick addendum to that, I've just had a look at rpcinfo on both
>>>>>>>> the
>>>>>>>> Ubuntu and Solaris NFS servers, does this indicate that nfs-rdma is
>>>>>>>> not actually running?
>>>>>>>>
>>>>>>>> rpcinfo -p
>>>>>>>> program vers proto port
>>>>>>>> 100000 2 tcp 111 portmapper
>>>>>>>> 100000 2 udp 111 portmapper
>>>>>>>> 100024 1 udp 37031 status
>>>>>>>> 100024 1 tcp 58463 status
>>>>>>>> 100021 1 udp 34989 nlockmgr
>>>>>>>> 100021 3 udp 34989 nlockmgr
>>>>>>>> 100021 4 udp 34989 nlockmgr
>>>>>>>> 100021 1 tcp 47979 nlockmgr
>>>>>>>> 100021 3 tcp 47979 nlockmgr
>>>>>>>> 100021 4 tcp 47979 nlockmgr
>>>>>>>> 100003 2 udp 2049 nfs
>>>>>>>> 100003 3 udp 2049 nfs
>>>>>>>> 100003 4 udp 2049 nfs
>>>>>>>> 100003 2 tcp 2049 nfs
>>>>>>>> 100003 3 tcp 2049 nfs
>>>>>>>> 100003 4 tcp 2049 nfs
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> Hi Ross:
>>>>>>>
>>>>>>> No, although that would be very nice, the Linux network maintainer
>>>>>>> didn't
>>>>>>> want RDMA transports sharing the network port space unfortunately.
>>>>>>>
>>>>>>> You would need to do this on the server to see if it is listening:
>>>>>>>
>>>>>>> # cat /proc/fs/nfsd/portlist
>>>>>>>
>>>>>>> You should see something like this:
>>>>>>>
>>>>>>> rdma 20049
>>>>>>> tcp 2049
>>>>>>> udp 2049
>>>>>>>
>>>>>>> The top line indicates that the rdma transport is listening on port
>>>>>>> 20049.
>>>>>>>
>>>>>>> If it's not showing, do this:
>>>>>>>
>>>>>>> # echo 20049 > /proc/fs/nfsd/portlist
>>>>>>>
>>>>>>> and repeat the 'cat' step above.
>>>>>>>
>>>>>>> To give us a little more detail to help debug, do this:
>>>>>>>
>>>>>>> # echo 32767 > /proc/sys/sunrpc/rpc_debug
>>>>>>>
>>>>>>> on both the client and server, then try the mount again. The dmesg
>>>>>>> log
>>>>>>> should have a detail trace on what is happening.
>>>>>>>
>>>>>>> Turn off the debug output as follows:
>>>>>>>
>>>>>>> # echo 0 > /proc/sys/sunrpc/rpc_debug
>>>>>>>
>>>>>>> Tom
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Jan 26, 2010 at 12:24 PM, Ross Smith
>>>>>>>> <myxiplx at googlemail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hey everyone,
>>>>>>>>>
>>>>>>>>> It's taken me a week, but I've finally gotten the 2.7.00 firmware
>>>>>>>>> for
>>>>>>>>> this system. I've also taken the step of installing a Ubuntu 9.10
>>>>>>>>> server for testing in addition to the Solaris server I already
>>>>>>>>> have.
>>>>>>>>>
>>>>>>>>> So far I'm still having no joy, nfs mounts fine over TCP, but if I
>>>>>>>>> try
>>>>>>>>> to use RDMA it fails.
>>>>>>>>>
>>>>>>>>> Machines in use:
>>>>>>>>> ============
>>>>>>>>> Solaris Server, build 129 (about 4 weeks old), using built in
>>>>>>>>> Infiniband
>>>>>>>>> drivers
>>>>>>>>> Solaris Client, same build
>>>>>>>>> Ubuntu 9.10 Server, using kernel drivers
>>>>>>>>> Ubuntu 9.10 Client
>>>>>>>>> CentOS 5.2 Client, with OFED 1.4.2 and nfs-utils 1.1.6
>>>>>>>>>
>>>>>>>>> All five machines are on identical hardware, with Mellanox ConnectX
>>>>>>>>> infiniband cards running firmware 2.7.00.
>>>>>>>>>
>>>>>>>>> They all seem to be running Infiniband fine, ipoib works perfectly
>>>>>>>>> and
>>>>>>>>> I can connect regular tcp nfs mounts over the infiniband links
>>>>>>>>> without
>>>>>>>>> any issues.
>>>>>>>>>
>>>>>>>>> With regular tcp nfs I'm getting consistent speeds of 300MB/s.
>>>>>>>>>
>>>>>>>>> However, nfs-rdma just does not want to work, no matter which
>>>>>>>>> combination of servers and clients I try:
>>>>>>>>>
>>>>>>>>> Ubuntu Client -> Solaris
>>>>>>>>> =================
>>>>>>>>> Commands used:
>>>>>>>>> # modprobe xprtrdma
>>>>>>>>> # mount -o proto=rdma,port=20049 192.168.101.1:/test/rdma ./nfstest
>>>>>>>>>
>>>>>>>>> This is the entire dmesg log, from first loading the driver, to
>>>>>>>>> attempting to connect nfs-rdma:
>>>>>>>>>
>>>>>>>>> [ 46.834146] mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0
>>>>>>>>> (April
>>>>>>>>> 4, 2008)
>>>>>>>>> [ 47.028093] ADDRCONF(NETDEV_UP): ib0: link is not ready
>>>>>>>>> [ 52.018562] ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
>>>>>>>>> [ 52.018698] ib0: multicast join failed for
>>>>>>>>> ff12:401b:ffff:0000:0000:0000:0000:00fb, status -22
>>>>>>>>> [ 54.014289] ib0: multicast join failed for
>>>>>>>>> ff12:401b:ffff:0000:0000:0000:0000:00fb, status -22
>>>>>>>>> [ 58.006864] ib0: multicast join failed for
>>>>>>>>> ff12:401b:ffff:0000:0000:0000:0000:00fb, status -22
>>>>>>>>> [ 62.027202] ib0: no IPv6 routers present
>>>>>>>>> [ 65.120791] RPC: Registered udp transport module.
>>>>>>>>> [ 65.120795] RPC: Registered tcp transport module.
>>>>>>>>> [ 65.129162] RPC: Registered rdma transport module.
>>>>>>>>> [ 65.992081] ib0: multicast join failed for
>>>>>>>>> ff12:401b:ffff:0000:0000:0000:0000:00fb, status -22
>>>>>>>>> [ 81.962465] ib0: multicast join failed for
>>>>>>>>> ff12:401b:ffff:0000:0000:0000:0000:00fb, status -22
>>>>>>>>> [ 83.593144] rpcrdma: connection to 192.168.101.1:20049 on
>>>>>>>>> mlx4_0,
>>>>>>>>> memreg 5 slots 32 ird 4
>>>>>>>>> [ 148.476967] rpcrdma: connection to 192.168.101.1:20049 closed
>>>>>>>>> (-111)
>>>>>>>>> [ 148.480488] rpcrdma: connection to 192.168.101.1:20049 closed
>>>>>>>>> (-111)
>>>>>>>>> [ 148.484421] rpcrdma: connection to 192.168.101.1:20049 closed
>>>>>>>>> (-111)
>>>>>>>>> [ 148.488376] rpcrdma: connection to 192.168.101.1:20049 closed
>>>>>>>>> (-111)
>>>>>>>>> [ 4311.663188] svc: failed to register lockdv1 RPC service (errno
>>>>>>>>> 97).
>>>>>>>>>
>>>>>>>>> At this point, the attempt crashed the Solaris server, and hung the
>>>>>>>>> mount attempt on the Ubuntu client, requiring ctrl-c on the client,
>>>>>>>>> and automatically rebooting the server.
>>>>>>>>>
>>>>>>>>> I then tried again, connecting to the Ubuntu nfs server. This time
>>>>>>>>> neither device hung or crashed, but I had very similar messages in
>>>>>>>>> the
>>>>>>>>> client log:
>>>>>>>>>
>>>>>>>>> # mount -o proto=rdma,port=20049 192.168.101.5:/home/ross/nfsexport
>>>>>>>>> ./nfstest
>>>>>>>>>
>>>>>>>>> [ 4435.102852] rpcrdma: connection to 192.168.101.5:20049 closed
>>>>>>>>> (-111)
>>>>>>>>> [ 4435.107492] rpcrdma: connection to 192.168.101.5:20049 closed
>>>>>>>>> (-111)
>>>>>>>>> [ 4435.111471] rpcrdma: connection to 192.168.101.5:20049 closed
>>>>>>>>> (-111)
>>>>>>>>> [ 4435.115468] rpcrdma: connection to 192.168.101.5:20049 closed
>>>>>>>>> (-111)
>>>>>>>>>
>>>>>>>>> So it seems that it's not the server: both Solaris and Ubuntu have
>>>>>>>>> the same problem, although Ubuntu at least does not crash when
>>>>>>>>> clients
>>>>>>>>> attempt to connect.
>>>>>>>>>
>>>>>>>>> I also get the same error if I attempt to connect from the CentOS
>>>>>>>>> 5.2
>>>>>>>>> machine which is using regular OFED to the Ubuntu server:
>>>>>>>>>
>>>>>>>>> CentOS 5.2 -> Ubuntu
>>>>>>>>> ================
>>>>>>>>> This time I'm running mount.rnfs directly as per the instructions
>>>>>>>>> in
>>>>>>>>> the OFED nfs-rdma release notes.
>>>>>>>>>
>>>>>>>>> commands used:
>>>>>>>>> # modprobe xprtrdma
>>>>>>>>> # mount.rnfs 192.168.101.5:/home/ross/nfsexport ./rdmatest -i -o
>>>>>>>>> proto=rdma,port=20049
>>>>>>>>>
>>>>>>>>> dmesg results look very similar:
>>>>>>>>> rpcrdma: connection to 192.168.101.5:20049 closed (-111)
>>>>>>>>> rpcrdma: connection to 192.168.101.5:20049 closed (-111)
>>>>>>>>> rpcrdma: connection to 192.168.101.5:20049 closed (-111)
>>>>>>>>> rpcrdma: connection to 192.168.101.5:20049 closed (-111)
>>>>>>>>>
>>>>>>>>> However attempting this has a bad effect on CentOS - the client
>>>>>>>>> crashes and I loose my ssh session.
>>>>>>>>>
>>>>>>>>> Does anybody have any ideas?
>>>>>>>>>
>>>>>>>>> thanks,
>>>>>>>>>
>>>>>>>>> Ross
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jan 18, 2010 at 6:31 PM, David Brean <David.Brean at sun.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I agree, update the HCA firmware before proceeding. [The
>>>>>>>>>> description
>>>>>>>>>> in
>>>>>>>>>> Bugzilla Bug 1711 seems to match the problem that you are
>>>>>>>>>> observing.]
>>>>>>>>>>
>>>>>>>>>> Also, if you want to help diagnose the "ib0: post_send failed",
>>>>>>>>>> take
>>>>>>>>>> a
>>>>>>>>>> look
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> http://lists.openfabrics.org/pipermail/general/2009-July/061118.html.
>>>>>>>>>>
>>>>>>>>>> -David
>>>>>>>>>>
>>>>>>>>>> Ross Smith wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Tom,
>>>>>>>>>>
>>>>>>>>>> No, you're right - I'm just using the support that's built into
>>>>>>>>>> the
>>>>>>>>>> kernel, and I agree, diagnostics from Solaris is proving very
>>>>>>>>>> tricky.
>>>>>>>>>> I do have a Solaris client connected to this and showing some
>>>>>>>>>> decent
>>>>>>>>>> speeds (over 900Mb/s), but I've been thinking that I might need to
>>>>>>>>>> get
>>>>>>>>>> a Linux server running for testing before I spend much more time
>>>>>>>>>> trying to get the two separate systems working.
>>>>>>>>>>
>>>>>>>>>> However, I have found over the weekend that I'm running older
>>>>>>>>>> firmware
>>>>>>>>>> and need that updating. I'd missed that in the nfs-rdma readme so
>>>>>>>>>> I'm
>>>>>>>>>> pretty sure that's going to be causing problems. I'm trying to
>>>>>>>>>> get
>>>>>>>>>> that resolved before I do too much other testing.
>>>>>>>>>>
>>>>>>>>>> Regular NFS running over the ipoib link seems fine, and I don't
>>>>>>>>>> get
>>>>>>>>>> any extra warnings using that. I can also run a full virtual
>>>>>>>>>> machine
>>>>>>>>>> quite happily over NFS, so despite the warnings, the link does
>>>>>>>>>> appear
>>>>>>>>>> stable and reliable.
>>>>>>>>>>
>>>>>>>>>> Ross
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Jan 18, 2010 at 4:30 PM, Tom Tucker
>>>>>>>>>> <tom at opengridcomputing.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Ross:
>>>>>>>>>>
>>>>>>>>>> I would check that you have IB RDMA actually working. The core
>>>>>>>>>> transport
>>>>>>>>>> issues suggest that there may be network problems that will
>>>>>>>>>> prevent
>>>>>>>>>> NFSRDMA
>>>>>>>>>> from working properly.
>>>>>>>>>>
>>>>>>>>>> The first question is whether or not you are actually using OFED.
>>>>>>>>>> You're
>>>>>>>>>> not
>>>>>>>>>> -- right? You're just using the support built into the 2.6.31
>>>>>>>>>> kernel?
>>>>>>>>>>
>>>>>>>>>> Second I don't think the mount is actually completing. I think the
>>>>>>>>>> command
>>>>>>>>>> is returning, but the mount never actually finishes. It's sitting
>>>>>>>>>> there
>>>>>>>>>> hung
>>>>>>>>>> trying to perform the first RPC to the server (RPC_NOP) and it's
>>>>>>>>>> never
>>>>>>>>>> succeeding. That's why you see all those connect/disconnect
>>>>>>>>>> messages
>>>>>>>>>> in
>>>>>>>>>> your
>>>>>>>>>> log file. It tries to send, gets an error, disconnects,
>>>>>>>>>> reconnects,
>>>>>>>>>> tries to
>>>>>>>>>> send .... you get the picture.
>>>>>>>>>>
>>>>>>>>>> Step 1 I think would be to ensure that you actually have IB up and
>>>>>>>>>> running.
>>>>>>>>>> IPoIB between the two seems a little dodgy given the dmesg log. Do
>>>>>>>>>> you
>>>>>>>>>> have
>>>>>>>>>> another Linux box you can use to test out
>>>>>>>>>> connectivity/configuration
>>>>>>>>>> with
>>>>>>>>>> your victim? There are test programs in OFED (rping) that would
>>>>>>>>>> help
>>>>>>>>>> you
>>>>>>>>>> do
>>>>>>>>>> this, but I don't believe they are available on Solaris.
>>>>>>>>>>
>>>>>>>>>> Tom
>>>>>>>>>>
>>>>>>>>>> Steve Wise wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> nfsrdma hang on ewg...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -------- Original Message --------
>>>>>>>>>> Subject: [ewg] nfs-rdma hanging with Ubuntu 9.10
>>>>>>>>>> Date: Fri, 15 Jan 2010 13:28:31 +0000
>>>>>>>>>> From: Ross Smith <myxiplx at googlemail.com>
>>>>>>>>>> To: ewg at openfabrics.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi folks, it's me again I'm afraid.
>>>>>>>>>>
>>>>>>>>>> Thanks to the help from this list, I have ipoib working, however I
>>>>>>>>>> seem to be having a few problems, not least of which is commands
>>>>>>>>>> hanging if I attempt to use nfs-rdma.
>>>>>>>>>>
>>>>>>>>>> Although the rmda mount command completes, the system then becomes
>>>>>>>>>> unresponsive if I attempt any command such as 'ls', even outside
>>>>>>>>>> of
>>>>>>>>>> the mounted folder. Umount also fails with the error "device is
>>>>>>>>>> busy".
>>>>>>>>>>
>>>>>>>>>> If anybody can spare the time to help it would be very much
>>>>>>>>>> appreciated. I do seem to have a lot of warnings in the logs, but
>>>>>>>>>> although I've tried searching for solutions haven't found anything
>>>>>>>>>> yet.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> System details
>>>>>>>>>> ============
>>>>>>>>>> - Ubuntu 9.10
>>>>>>>>>> (kernel 2.6.31)
>>>>>>>>>> - Mellanox ConnectX QDR card
>>>>>>>>>> - Flextronics DDR switch
>>>>>>>>>> - OpenSolaris NFS server, running one of the latest builds for
>>>>>>>>>> troubleshooting
>>>>>>>>>> - OpenSM running on another Ubuntu 9.10 box with a Mellanox
>>>>>>>>>> Infinihost III Lx card
>>>>>>>>>>
>>>>>>>>>> I am using the kernel drivers only, I have not installed OFED on
>>>>>>>>>> this
>>>>>>>>>> machine.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Loading driver
>>>>>>>>>> ============
>>>>>>>>>> The driver appears to load, and ipoib works, but there are rather
>>>>>>>>>> a
>>>>>>>>>> lot of warnings from dmesg.
>>>>>>>>>>
>>>>>>>>>> I am loading the driver with:
>>>>>>>>>> $ sudo modprobe mlx4_ib
>>>>>>>>>> $ sudo modprobe ib_ipoib
>>>>>>>>>> $ sudo ifconfig ib0 192.168.101.4 netmask 255.255.255.0 up
>>>>>>>>>>
>>>>>>>>>> And that leaves me with:
>>>>>>>>>> $ lsmod
>>>>>>>>>> Module Size Used by
>>>>>>>>>> ib_ipoib 72452 0
>>>>>>>>>> ib_cm 37196 1 ib_ipoib
>>>>>>>>>> ib_sa 19812 2 ib_ipoib,ib_cm
>>>>>>>>>> mlx4_ib 42720 0
>>>>>>>>>> ib_mad 37524 3 ib_cm,ib_sa,mlx4_ib
>>>>>>>>>> ib_core 57884 5
>>>>>>>>>> ib_ipoib,ib_cm,ib_sa,mlx4_ib,ib_mad
>>>>>>>>>> binfmt_misc 8356 1
>>>>>>>>>> ppdev 6688 0
>>>>>>>>>> psmouse 56180 0
>>>>>>>>>> serio_raw 5280 0
>>>>>>>>>> mlx4_core 84728 1 mlx4_ib
>>>>>>>>>> joydev 10272 0
>>>>>>>>>> lp 8964 0
>>>>>>>>>> parport 35340 2 ppdev,lp
>>>>>>>>>> iptable_filter 3100 0
>>>>>>>>>> ip_tables 11692 1 iptable_filter
>>>>>>>>>> x_tables 16544 1 ip_tables
>>>>>>>>>> usbhid 38208 0
>>>>>>>>>> e1000e 122124 0
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> At this point I can ping the Solaris server over the IP link.
>>>>>>>>>> Although I do need to issue a ping from Solaris before I get a
>>>>>>>>>> reply.
>>>>>>>>>> I'm mentioning that it in case it's relevant, but at this point
>>>>>>>>>> I'm
>>>>>>>>>> assuming that's just a firewall setting on the server.
>>>>>>>>>>
>>>>>>>>>> But although ping works, I am starting to get some dmesg warnings,
>>>>>>>>>> I
>>>>>>>>>> just don't know if they are relevant:
>>>>>>>>>> [ 313.692072] mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0
>>>>>>>>>> (April
>>>>>>>>>> 4,
>>>>>>>>>> 2008)
>>>>>>>>>> [ 313.885220] ADDRCONF(NETDEV_UP): ib0: link is not ready
>>>>>>>>>> [ 316.880450] ib0: multicast join failed for
>>>>>>>>>> ff12:401b:ffff:0000:0000:0000:0000:00fb, status -22
>>>>>>>>>> [ 316.880573] ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
>>>>>>>>>> [ 316.880789] ib0: multicast join failed for
>>>>>>>>>> ff12:401b:ffff:0000:0000:0000:0000:00fb, status -22
>>>>>>>>>> [ 320.873613] ib0: multicast join failed for
>>>>>>>>>> ff12:401b:ffff:0000:0000:0000:0000:00fb, status -22
>>>>>>>>>> [ 327.147114] ib0: no IPv6 routers present
>>>>>>>>>> [ 328.861550] ib0: multicast join failed for
>>>>>>>>>> ff12:401b:ffff:0000:0000:0000:0000:00fb, status -22
>>>>>>>>>> [ 344.834440] ib0: multicast join failed for
>>>>>>>>>> ff12:401b:ffff:0000:0000:0000:0000:00fb, status -22
>>>>>>>>>> [ 360.808312] ib0: multicast join failed for
>>>>>>>>>> ff12:401b:ffff:0000:0000:0000:0000:00fb, status -22
>>>>>>>>>> [ 376.782186] ib0: multicast join failed for
>>>>>>>>>> ff12:401b:ffff:0000:0000:0000:0000:00fb, status -22
>>>>>>>>>>
>>>>>>>>>> And at this point however, regular nfs mounts work fine over the
>>>>>>>>>> ipoib
>>>>>>>>>> link:
>>>>>>>>>> $ sudo mount 192.168.100.1:/test/rdma ./nfstest
>>>>>>>>>>
>>>>>>>>>> Bug again, that again adds warnings to dmesg:
>>>>>>>>>> [ 826.456902] RPC: Registered udp transport module.
>>>>>>>>>> [ 826.456905] RPC: Registered tcp transport module.
>>>>>>>>>> [ 841.553135] svc: failed to register lockdv1 RPC service (errno
>>>>>>>>>> 97).
>>>>>>>>>>
>>>>>>>>>> And the speed is definitely nothing to write home about, copying a
>>>>>>>>>> 100mb file takes over 10 seconds:
>>>>>>>>>> $ time cp ./100mb ./100mb2
>>>>>>>>>>
>>>>>>>>>> real 0m10.472s
>>>>>>>>>> user 0m0.000s
>>>>>>>>>> sys 0m1.248s
>>>>>>>>>>
>>>>>>>>>> And again with warnings appearing in dmesg:
>>>>>>>>>> [ 872.373364] ib0: post_send failed
>>>>>>>>>> [ 872.373407] ib0: post_send failed
>>>>>>>>>> [ 872.373448] ib0: post_send failed
>>>>>>>>>>
>>>>>>>>>> I think this is a client issue rather than a problem on the server
>>>>>>>>>> as
>>>>>>>>>> the same test on an OpenSolaris client takes under half a second:
>>>>>>>>>> # time cp ./100mb ./100mb2
>>>>>>>>>>
>>>>>>>>>> real 0m0.334s
>>>>>>>>>> user 0m0.001s
>>>>>>>>>> sys 0m0.176s
>>>>>>>>>>
>>>>>>>>>> Although the system is definitely not right, my long term aim is
>>>>>>>>>> to
>>>>>>>>>> run nfs-rdma on this system, so my next test was to try that and
>>>>>>>>>> see
>>>>>>>>>> if the speed improved:
>>>>>>>>>>
>>>>>>>>>> $ sudo umount ./nfstest
>>>>>>>>>> $ sudo mount -o rdma,port=20049 192.168.101.1:/test/rdma ./nfstest
>>>>>>>>>>
>>>>>>>>>> That takes a long time to connect. It does eventually go through,
>>>>>>>>>> but
>>>>>>>>>> only after the following errors in dmesg:
>>>>>>>>>>
>>>>>>>>>> [ 1140.698659] RPC: Registered rdma transport module.
>>>>>>>>>> [ 1155.697672] rpcrdma: connection to 192.168.101.1:20049 on
>>>>>>>>>> mlx4_0,
>>>>>>>>>> memreg 5 slots 32 ird 4
>>>>>>>>>> [ 1160.688455] rpcrdma: connection to 192.168.101.1:20049 closed
>>>>>>>>>> (-103)
>>>>>>>>>> [ 1160.693818] rpcrdma: connection to 192.168.101.1:20049 on
>>>>>>>>>> mlx4_0,
>>>>>>>>>> memreg 5 slots 32 ird 4
>>>>>>>>>> [ 1160.695131] svc: failed to register lockdv1 RPC service (errno
>>>>>>>>>> 97).
>>>>>>>>>> [ 1170.676049] rpcrdma: connection to 192.168.101.1:20049 closed
>>>>>>>>>> (-103)
>>>>>>>>>> [ 1170.681458] rpcrdma: connection to 192.168.101.1:20049 on
>>>>>>>>>> mlx4_0,
>>>>>>>>>> memreg 5 slots 32 ird 4
>>>>>>>>>> [ 1190.647355] rpcrdma: connection to 192.168.101.1:20049 closed
>>>>>>>>>> (-103)
>>>>>>>>>> [ 1190.652778] rpcrdma: connection to 192.168.101.1:20049 on
>>>>>>>>>> mlx4_0,
>>>>>>>>>> memreg 5 slots 32 ird 4
>>>>>>>>>> [ 1220.602353] rpcrdma: connection to 192.168.101.1:20049 closed
>>>>>>>>>> (-103)
>>>>>>>>>> [ 1220.607809] rpcrdma: connection to 192.168.101.1:20049 on
>>>>>>>>>> mlx4_0,
>>>>>>>>>> memreg 5 slots 32 ird 4
>>>>>>>>>> [ 1250.557397] rpcrdma: connection to 192.168.101.1:20049 closed
>>>>>>>>>> (-103)
>>>>>>>>>> [ 1250.562817] rpcrdma: connection to 192.168.101.1:20049 on
>>>>>>>>>> mlx4_0,
>>>>>>>>>> memreg 5 slots 32 ird 4
>>>>>>>>>> [ 1281.522735] rpcrdma: connection to 192.168.101.1:20049 closed
>>>>>>>>>> (-103)
>>>>>>>>>> [ 1281.528442] rpcrdma: connection to 192.168.101.1:20049 on
>>>>>>>>>> mlx4_0,
>>>>>>>>>> memreg 5 slots 32 ird 4
>>>>>>>>>> [ 1311.477845] rpcrdma: connection to 192.168.101.1:20049 closed
>>>>>>>>>> (-103)
>>>>>>>>>> [ 1311.482983] rpcrdma: connection to 192.168.101.1:20049 on
>>>>>>>>>> mlx4_0,
>>>>>>>>>> memreg 5 slots 32 ird 4
>>>>>>>>>> [ 1341.432758] rpcrdma: connection to 192.168.101.1:20049 closed
>>>>>>>>>> (-103)
>>>>>>>>>> [ 1341.438212] rpcrdma: connection to 192.168.101.1:20049 on
>>>>>>>>>> mlx4_0,
>>>>>>>>>> memreg 5 slots 32 ird 4
>>>>>>>>>>
>>>>>>>>>> However, at this point my shell session becomes unresponsive if I
>>>>>>>>>> attempt so much as a 'ls'. The system hasn't hung completely
>>>>>>>>>> however
>>>>>>>>>> as I can still connect another ssh session and restart with
>>>>>>>>>> $ sudo init 6
>>>>>>>>>>
>>>>>>>>>> Can anybody help? Is there anything obvious I am doing wrong
>>>>>>>>>> here?
>>>>>>>>>>
>>>>>>>>>> thanks,
>>>>>>>>>>
>>>>>>>>>> Ross
>>>>>>>>>> _______________________________________________
>>>>>>>>>> ewg mailing list
>>>>>>>>>> ewg at lists.openfabrics.org
>>>>>>>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> ewg mailing list
>>>>>>>>>> ewg at lists.openfabrics.org
>>>>>>>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> ewg mailing list
>>>>>>>> ewg at lists.openfabrics.org
>>>>>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>
>
More information about the ewg
mailing list