[ofa-general] Low NFS RDMA performance with Connect X

[Kuchimanchi, Ramachandra]

Hi,

I took the NFS RDMA code from the Mellanox NFS RDMA SDK, compiled it with
OFED-1.2.5 and tried it out with Connect X HCAs and also MT25208. I found
that the Iozone read and write performance numbers are very low on Connect X.

For a 128 MB file and a 128 KB record size 

NFS RDMA SDK on MT25028:              Read: 861 MB/s       Write: 185 MB/s
OFED-1.2.5 with NFS RDMA modules      Read: 849 MB/s       Write: 184 MB/s  
on a MT25208
OFED-1.2.5 with NFS RDMA modules      Read: 451 MB/s       Write: 79 MB/s
on Connect X

Has any one tried this out or know of a reason why the numbers are so low
on Connect X ?

Test-setup:
Server and single client running RHEL 5
MT25208 tests were with dual processor 64-bit AMD machines
Connect X tests were with dual processor dual core 64-bit AMD machines
Connect X HCA FW ver: 2.1
NFS mount was in async mode and iozone tests were run with -c option.

More Iozone results for a record size of 64 KB (values below in KB/sec):

Read test			

File Size  SDK on MT25208      OFED-1.2.5 on MT25208   OFED-1.2.5 on ConnectX
(in MB) 	
64           1684819               1701916                   459279
128          882580                 870180                   462486
256          922081                 921932                   468063
512          871136                 909221                   452969
1024         900314                 910171                   442215
2048         908117                 849710                   676776
			
Write test			

File Size  SDK on MT25208    OFED-1.2.5 on MT25208   OFED-1.2.5 on ConnectX
(in MB)	
64           184154                 182483                   78424
128          190126                 189284                   81869
256          194921                 173124                   85813
512          199666                 192110                   87628
1024         208924                 199240                   126415
2048         180128                 195278                   123020

Regards,
Ram
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20070904/e6c2ceed/attachment.html>