[nfs-rdma-devel] [ofa-general] Status of NFS-RDMA ?
James Lentini
jlentini at netapp.com
Mon Feb 4 08:09:43 PST 2008
On Mon, 4 Feb 2008, Pawel Dziekonski wrote:
> On Mon, 04 Feb 2008 at 09:59:50AM -0500, James Lentini wrote:
> >
> >
> > On Fri, 1 Feb 2008, Pawel Dziekonski wrote:
> >
> > > On Fri, 01 Feb 2008 at 10:56:55AM -0500, James Lentini wrote:
> > > > > # mount 10.2.2.1:/scratch /mnt -i -o rdma,port=2050 Unsupported nfs
> > > > > mount option: rdma
> > > > >
> > > > > looks like I definitelly need a rdma-enabled mount, which comes in
> > > > > http://www.mellanox.com/downloads/NFSoRDMA/OFED-1.2-NFS-RDMA.gz, so:
> > > >
> > > > If you are using the NFS/RDMA client in 2.6.24, you need the version
> > > > of the mount.nfs command in nfs-utils-1.1.1 or greater.
> > > >
> > > > The "Unsupported nfs mount option: rdma" error message makes me
> > > > suspect you are not using the correct version of mount.nfs. What is
> > > > the output of "mount.nfs -V" and "mount -V"? It may be that the
> > > > version of mount you are using does not automatically invoke mount.nfs
> > > > for nfs mounts. I'd suggest specifying mount.nfs in the command above:
> > >
> > > # mount.nfs -V
> > > mount.nfs (linux nfs-utils 1.1.1)
> > >
> > > > mount.nfs 10.2.2.1:/scratch /mnt -i -o rdma,port=2050
> > >
> > > # mount.nfs 10.2.2.1:/scratch /mnt -i -o rdma,port=2050 -v
> > > mount.nfs: timeout set for Fri Feb 1 23:48:08 2008
> > > mount.nfs: text-based options: 'rdma,port=2050,addr=10.2.2.1'
> > > mount.nfs: internal error
> > >
> > > :(
> >
> > I'm going to guess that the "internal error" message means that your
> > kernel does not support the NFS string mount API. Are you sure your
> > kernel you are using is an exact copy of Tom Tucker's git tree? It
> > can NOT have the OFED 1.2 package installed.
>
> kernel was pulled from Tom's tree, Jan 25th.
>
> # cat .git/config
> [core]
> repositoryformatversion = 0
> filemode = true
> bare = false
> logallrefupdates = true
> [remote "origin"]
> url = git://git.linux-nfs.org/projects/tomtucker/xprt-switch-2.6.git
> fetch = +refs/heads/*:refs/remotes/origin/*
> [branch "merged-fixes"]
> remote = origin
> merge = refs/heads/merged-fixes
>
>
> config: https://cefeid.wcss.wroc.pl/d/tmp/c-2.6.24-rc6
>
> What do you mean 'It can NOT have the OFED 1.2 package installed.'?
> I can not install OFED 1.2 on this machine at all?!
I'm refering to the OFED 1.2 NFS/RDMA package you cited above:
http://www.mellanox.com/downloads/NFSoRDMA/OFED-1.2-NFS-RDMA.gz, so
You can NOT install that on Tom Tucker's kernel. That is an old
version of the NFS/RDMA software that will conflict with the latest
code.
If the OFED version you want to use is supported on Tom Tucker's
kernel (2.6.24-rc6), there should be no problem using OFED. The OFED
1.2 NFS/RDMA release is a different story.
> > If the kernel is correct, can you send the output of:
> >
> > strace -e trace=all 10.2.2.1:/scratch /mnt -i -o rdma,port=2050
>
> execve("/sbin/mount.nfs", ["mount.nfs", "10.2.2.1:/scratch", "/mnt", "-i", "-o", "rdma,port=2050"], [/* 28 vars */]) = 0
> uname({sys="Linux", node="ib2", ...}) = 0
> brk(0) = 0x613000
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b68b8a78000
> access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
> open("/opt/intel/mkl/10.0.1.014/lib/em64t/tls/x86_64/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
> stat("/opt/intel/mkl/10.0.1.014/lib/em64t/tls/x86_64", 0x7ffff202db10) = -1 ENOENT (No such file or directory)
> open("/opt/intel/mkl/10.0.1.014/lib/em64t/tls/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
> stat("/opt/intel/mkl/10.0.1.014/lib/em64t/tls", 0x7ffff202db10) = -1 ENOENT (No such file or directory)
> open("/opt/intel/mkl/10.0.1.014/lib/em64t/x86_64/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
> stat("/opt/intel/mkl/10.0.1.014/lib/em64t/x86_64", 0x7ffff202db10) = -1 ENOENT (No such file or directory)
> open("/opt/intel/mkl/10.0.1.014/lib/em64t/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
> stat("/opt/intel/mkl/10.0.1.014/lib/em64t", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> open("/opt/intel/fce/10.1.011/lib/tls/x86_64/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
> stat("/opt/intel/fce/10.1.011/lib/tls/x86_64", 0x7ffff202db10) = -1 ENOENT (No such file or directory)
> open("/opt/intel/fce/10.1.011/lib/tls/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
> stat("/opt/intel/fce/10.1.011/lib/tls", 0x7ffff202db10) = -1 ENOENT (No such file or directory)
> open("/opt/intel/fce/10.1.011/lib/x86_64/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
> stat("/opt/intel/fce/10.1.011/lib/x86_64", 0x7ffff202db10) = -1 ENOENT (No such file or directory)
> open("/opt/intel/fce/10.1.011/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
> stat("/opt/intel/fce/10.1.011/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> open("/opt/intel/cce/10.1.011/lib/tls/x86_64/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
> stat("/opt/intel/cce/10.1.011/lib/tls/x86_64", 0x7ffff202db10) = -1 ENOENT (No such file or directory)
> open("/opt/intel/cce/10.1.011/lib/tls/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
> stat("/opt/intel/cce/10.1.011/lib/tls", 0x7ffff202db10) = -1 ENOENT (No such file or directory)
> open("/opt/intel/cce/10.1.011/lib/x86_64/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
> stat("/opt/intel/cce/10.1.011/lib/x86_64", 0x7ffff202db10) = -1 ENOENT (No such file or directory)
> open("/opt/intel/cce/10.1.011/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
> stat("/opt/intel/cce/10.1.011/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> open("/etc/ld.so.cache", O_RDONLY) = 3
> fstat(3, {st_mode=S_IFREG|0644, st_size=38220, ...}) = 0
> mmap(NULL, 38220, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2b68b8a79000
> close(3) = 0
> open("/lib64/tls/libc.so.6", O_RDONLY) = 3
> read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\304"..., 832) = 832
> fstat(3, {st_mode=S_IFREG|0755, st_size=1622600, ...}) = 0
> mmap(0x33f2600000, 2314184, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x33f2600000
> mprotect(0x33f272c000, 1085384, PROT_NONE) = 0
> mmap(0x33f282c000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x12c000) = 0x33f282c000
> mmap(0x33f2831000, 16328, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x33f2831000
> close(3) = 0
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b68b8a83000
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b68b8a84000
> mprotect(0x33f282c000, 12288, PROT_READ) = 0
> mprotect(0x33f2514000, 4096, PROT_READ) = 0
> arch_prctl(ARCH_SET_FS, 0x2b68b8a83b00) = 0
> munmap(0x2b68b8a79000, 38220) = 0
> getuid() = 0
> uname({sys="Linux", node="ib2", ...}) = 0
> uname({sys="Linux", node="ib2", ...}) = 0
> brk(0) = 0x613000
> brk(0x634000) = 0x634000
> lstat("/mnt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> stat("/mnt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> access("/mnt", X_OK) = 0
> mount("10.2.2.1:/scratch", "/mnt", "nfs", 0, "rdma,port=2050,addr=10.2.2.1") = -1 EIO (Input/output error)
The above makes me suspect that the OFED 1.2 NFS/RDMA package is
installed.
If that is not the case, can you turn on NFS mount debugging?
dmesc -c > /dev/null
echo 1024 > /proc/sys/sunrpc/nfs_debug
mount.nfs 10.2.2.1:/scratch /mnt -i -o rdma,port=2050
dmesg > output
Please send the contents of "output".
> write(2, "mount.nfs: internal error\n", 26mount.nfs: internal error
> ) = 26
> exit_group(32) = ?
> Process 32607 detached
>
>
> --
> Pawel Dziekonski <pawel.dziekonski at wcss.pl>
> Wroclaw Centre for Networking & Supercomputing, HPC Department
> Politechnika Wr., pl. Grunwaldzki 9, bud. D2/101, 50-377 Wroclaw, POLAND
> phone: +48 71 3202043, fax: +48 71 3225797, http://www.wcss.wroc.pl
More information about the general
mailing list