[ofa-general] AMP/RDMA performance expectation

[Talpey, Thomas]

At 03:36 AM 3/11/2008, yangdong wrote:
>
>Hello! I implemented a message passing subsystem -- AMP for our cluster
>file system--DCFS3,
>I want AMP to run over Infiniband, I designed it in kernel(verbs).
>when client writing file, my protocol is implemented by RDMA WRITE, and
>reading file by RDMA
>READ, but perf of AMP is less than my expectation. Single thread (1t)
>read is 215MB/s, 4t is 340MB/s.
>And I see that the read file perf of NFS/RDMA is close to 600MB/s (1t).

NFS/RDMA is not single threaded. The client has several ways to keep multiple
I/Os active, among them use of a kernel rpc state machine, enlisting application
threads performing synch i/o, use of kernel helpers (rpciod's), the filesystem buffer
cache, and the RPC slot table. Also, its protocol supports credits and RDMA
scatter/gather, and is tuned to optimize server transfers.

BTW, NFS/RDMA achieves much more than 600MB/s. On DDR IB, for example,
Sandia's testing achieved over double that.

Just some ideas, since you give so little information about your implementation
it's hard to conclude anything.

Tom.

>I made use of FMR and All physical reg ..and scatter/gather, but why the
>perf of AMP is lower?
>
>
>
>-- 
>Dong Yang <yangdong at ncic.ac.cn>
>Institute of Computing Technology, Chinese Academy of Sciences
>Address: National Research Center for Intelligent Computing Systems 
>(NCIC), P.O. Box 2704, Beijing 100080, P.R. China
>Phone: +86-10-62601005
>
>_______________________________________________
>general mailing list
>general at lists.openfabrics.org
>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
>To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general