[openib-general] A critique of RDMA PUT/GET in HPC

[Tom Tucker]

On Thu, 2006-08-24 at 15:53 -0700, Greg Lindahl wrote:
> For those of you interested in this topic, there's an interesting
> article by Patrick Geoffrey in HPCWire entitled "A Critique of RDMA".
> 
> http://www.hpcwire.com/hpc/815242.html
> 
> (you might have to be a subscriber, but I'm sure Patrick would send
> you a copy if you ask.)
> 
> It's basically a critique of why SEND/RECV is better for MPI
> implementations than PUT/GET.

He does say this, but his analysis does not support this conclusion. His
analysis revolves around MPI send/recv, not the MPI 2.0 get/put
services. He makes the point (true in my opinion) that the MPI_RECV
64bit (tag,communicator) filter make MPI_RECV prickly to implement on
IB/iWARP SEND/RECV and IB/iWARP RDMA. His data are drawn from
observations of MPI applications that use MPI send/recv mapped to an
RDMA transport. However, his conclusion covers a programming model (MPI
get/put) that is not observed in the data. In other words, he doesn't
compare the performance of an algorithm implemented using MPI send/recv
vs. the same algorithm implemented using MPI get/put. He evaluates the
performance of an algorithm implemented using MPI send/recv mapped to an
RDMA transport and then says because this mapping has problems that the
RDMA programming model is bad. That conclusion is not supported by his
analysis or his data. A valid conclusion IMO is that "MPI send/recv can
be most efficiently implemented over an unconnected reliable datagram
protocol that supports 64bit tag matching at the data sink." And not
coincidentally, Myricom has this ;-)

I DO agree that it is interesting reading. :-), it's definitely got
people fired up.

My 2 cents.


> 
> Even if you don't agree with him, it's good reading. For motivation,
> you might want to note that most of the SEND/RECV-based products
> mentioned achieve better MPI 0-byte latency than IB Verbs-based MPI
> implementations.
> 
> While I don't agree with everything Patrick says, this does get back
> to my point that I've run into many people who assume that PUT/GET is
> always the right way to do things. And it isn't.
> 
> -- greg
> 
> 
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>