[ofa-general] nfs/rdma slow with uncached data

[John Marshall]

Hi,

I have done some nfs/rdma tests and found impressive transfer
rates, but only when the file data is in cache. On the other hand,
using straight nfs over ipoib I am able to get decent transfer
rates one first read (not cached) when I tweak:
    echo 128 > /proc/fs/nfsd/pool_threads
    echo 1024 > /sys/block/sd?/queue/nr_requests
    echo 16384 > /sys/block/sd?/queue/read_ahead_kb (or using blockdev 
with 8192)

My question: Does the nfs/rdma setup bypass the readahead
mechanism in the kernel?

If it does,
1) this may account for the major difference described above,
2) explain why transfers are _very_ fast only on the second go
     around--because it is in cached (assuming it fits in the cache)

For both cases, I am using a 2.6.26 bigmem kernel with the necessary
tweaks.

Thanks,
John