[openib-general] NFS over IPoIB performance..

[Troy Benjegerdes]

On Thu, Mar 24, 2005 at 05:31:36PM -0800, Libor Michalek wrote:
> On Tue, Mar 22, 2005 at 08:57:53PM -0600, Troy Benjegerdes wrote:
> > Well, I'm quite impressed.. I've been running NFS over IPoIB to a server
> > with a 3ware SATA raid card in it, and nothing's crashed ;)
> > 
> > My only real problem is there seems to be no good way to set readahead
> > for the NFS client. I'm averaging 75mb/sec or so, and I've seen peaks of
> > 100MB/sec, with clients from two different machines running the GAMESS
> > computational chemistry code.
> 
>   I'm curious, where you running NFS over UDP or TCP? If the former
> could you try the later?
> 
> -Libor

With TCP:
10.0.0.10:/scratch/md1 on /nfs/da0/md1 type nfs
(rw,v3,rsize=32768,wsize=32768,hard,tcp,lock,addr=10.0.0.10)

opteron1:64bit:/nfs/da0/md1/troy$ dd if=junk of=/dev/null
20971520+0 records in
20971520+0 records out
10737418240 bytes transferred in 113.188032 seconds (94863547 bytes/sec)


With UDP:

opteron1:64bit:/nfs/da0$ mount  10.0.0.10:/scratch/md1 /nfs/da0/md1 -o
rsize=32768,wsize=32768,nfsvers=3
nfs warning: mount version older than kernel
opteron1:64bit:/nfs/da0$ cd md1/troy/
opteron1:64bit:/nfs/da0/md1/troy$ dd if=junk of=/dev/null
20971520+0 records in
20971520+0 records out
10737418240 bytes transferred in 96.758483 seconds (110971337 bytes/sec)

And one more time with tcp, in case there are buffer-cache effects.

opteron1:64bit:/nfs/da0$ mount  10.0.0.10:/scratch/md1 /nfs/da0/md1 -o
rsize=32768,wsize=32768,nfsvers=3,tcp
nfs warning: mount version older than kernel
opteron1:64bit:/nfs/da0$ cd md1/troy/
opteron1:64bit:/nfs/da0/md1/troy$ dd if=junk of=/dev/null
20971520+0 records in
20971520+0 records out
10737418240 bytes transferred in 129.839903 seconds (82697368 bytes/sec)

One last time with UDP:

opteron1:64bit:/nfs/da0$ mount  10.0.0.10:/scratch/md1 /nfs/da0/md1 -o
rsize=32768,wsize=32768,nfsvers=3
nfs warning: mount version older than kernel
opteron1:64bit:/nfs/da0$ cd troy/
opteron1:64bit:/nfs/da0/troy$ cd ../md1/troy/
opteron1:64bit:/nfs/da0/md1/troy$ dd if=junk of=/dev/null
20971520+0 records in
20971520+0 records out
10737418240 bytes transferred in 96.915829 seconds (110791172 bytes/sec)