[nfs-rdma-devel] [ofa-general] Status of NFS-RDMA ? (fwd)

[Pawel Dziekonski]

hi,

the saga continues. ;)

very basic benchmarks and surprising (at least for me) results - it
look's like reading is much slower than writing and NFS/RDMA is twice
slower in reading than classic NFS. :o

results below - comments appreciated!
regards, Pawel


both nfs server and client have 8-cores, 16 GB RAM, Mellanox DDR HCAs
(MT25204) connected port-port (no switch).

local_hdd - 2 sata2 disks in soft-raid0,
nfs_ipoeth - classic nfs over ethernet,
nfs_ipoib - classic nfs over IPoIB,
nfs_rdma - NFS/RDMA.

simple write of 36GB file with dd (both machines have 16GB RAM):
/usr/bin/time -p dd if=/dev/zero of=/mnt/qqq bs=1M count=36000

local_hdd    sys 54.52    user 0.04    real 254.59
             
nfs_ipoib    sys 36.35    user 0.00    real 266.63
nfs_rdma     sys 39.03    user 0.02    real 323.77
nfs_ipoeth   sys 34.21    user 0.01    real 375.24

remount /mnt to clear cache and read a file from nfs share and
write it to /dev/:
/usr/bin/time -p dd if=/mnt/qqq of=/scratch/qqq bs=1M

nfs_ipoib   sys 59.04    user 0.02    real 571.57
nfs_ipoeth  sys 58.92    user 0.02    real 606.61
nfs_rdma    sys 62.57    user 0.03    real 1296.36



results from bonnie++:

Version  1.03c      ------Sequential Write ------ --Sequential Read -- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr-  --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP  K/sec %CP  /sec %CP
local_hdd  35G:128k           93353  12 58329   6           143293   7 243.6   1
local_hdd  35G:256k           92283  11 58189   6           144202   8 172.2   2
local_hdd  35G:512k           93879  12 57715   6           144167   8 128.2   4
local_hdd 35G:1024k           93075  12 58637   6           144172   8  95.3   7
nfs_ipoeth 35G:128k           91325   7 31848   4            64299   4 170.2   1
nfs_ipoeth 35G:256k           90668   7 32036   5            64542   4 163.2   2
nfs_ipoeth 35G:512k           93348   7 31757   5            64454   4  85.7   3
nfs_ipoet 35G:1024k           91283   7 31869   5            64241   5  51.7   4
nfs_ipoib  35G:128k           91733   7 36641   5            65839   4 178.4   2
nfs_ipoib  35G:256k           92453   7 36567   6            66682   4 166.9   3
nfs_ipoib  35G:512k           91157   7 37660   6            66318   4  86.8   3
nfs_ipoib 35G:1024k           92111   7 35786   6            66277   5  53.3   4
nfs_rdma   35G:128k           91152   8 29942   5            32147   2 187.0   1
nfs_rdma   35G:256k           89772   7 30560   5            34587   2 158.4   3
nfs_rdma   35G:512k           91290   7 29698   5            34277   2  60.9   2
nfs_rdma  35G:1024k           91336   8 29052   5            31742   2  41.5   3
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
local_hdd        16 10587  36 +++++ +++  8674  29 10727  35 +++++ +++  7015  28
local_hdd        16 11372  41 +++++ +++  8490  29 11192  43 +++++ +++  6881  27
local_hdd        16 10789  35 +++++ +++  8520  29 11468  46 +++++ +++  6651  24
local_hdd        16 10841  40 +++++ +++  8443  28 11162  41 +++++ +++  6441  22
nfs_ipoeth       16  3753   7 13390  12  3795   7  3773   8 22181  16  3635   7
nfs_ipoeth       16  3762   8 12358   7  3713   8  3753   7 20448  13  3632   6
nfs_ipoeth       16  3834   7 12697   6  3729   8  3725   9 22807  11  3673   7
nfs_ipoeth       16  3729   8 14260  10  3774   7  3744   7 25285  14  3688   7
nfs_ipoib        16  6803  17 +++++ +++  6843  15  6820  14 +++++ +++  5834  11
nfs_ipoib        16  6587  16 +++++ +++  4959   9  6832  14 +++++ +++  5608  12
nfs_ipoib        16  6820  18 +++++ +++  6636  15  6479  15 +++++ +++  5679  13
nfs_ipoib        16  6475  14 +++++ +++  6435  14  5543  11 +++++ +++  5431  11
nfs_rdma         16  7014  15 +++++ +++  6714  10  7001  14 +++++ +++  5683   8
nfs_rdma         16  7038  13 +++++ +++  6713  12  6956  11 +++++ +++  5488   8
nfs_rdma         16  7058  12 +++++ +++  6797  11  6989  14 +++++ +++  5761   9
nfs_rdma         16  7201  13 +++++ +++  6821  12  7072  15 +++++ +++  5609   9


-- 
Pawel Dziekonski <pawel.dziekonski at wcss.pl>
Wroclaw Centre for Networking & Supercomputing, HPC Department
Politechnika Wr., pl. Grunwaldzki 9, bud. D2/101, 50-377 Wroclaw, POLAND
phone: +48 71 3202043, fax: +48 71 3225797, http://www.wcss.wroc.pl