[ofa-general] general questions about librdmacm

[Roland Dreier]

 > The RDMA library has features for joining a multicast group.  Could you
 > point me at the appropriate specifications about infiniband relating to
 > the limitations of RDMA when used with multicast (or hand me an
 > explanation would be fine too :)

RDMA (in the sense of remote direct memory access, ie one sided
operations) can only be done over reliable connected transport.
Multicast works only over unreliable datagram transport.

You can get kernel bypass and zero-copy send/receive for multicast with
libibverbs and librdmacm.  But that strictly speaking isn't RDMA.

 > Also to answer your questions about performance requirements: today
 > totem is entirely cpu bound when running on GIGE hardware with jumbo
 > frames.  A majority of this time I believe is spent copying message data
 > between kernel and userspace, and then the other copies that take place
 > around the protocol.  I'd like to remove any copies that are present if
 > possible to give the CPU more bandwidth to execute the protocol.

Seems implausible given today's memory bandwidths.  I've seen slightly
in excess of 10 gigabits/sec with TCP on IPoIB -- this is with netpipe
and using large send/large receive offload but still all data is being
copied at least once.

Profiling to find out where your CPU is really going probably wouldn't
be a bad idea before you go to the effort of reimplementing everything
on InfiniBand.

 - R.