[openib-general] RFC userspace / MPI multicast support

[Caitlin Bestler]

openib-general-bounces at openib.org wrote:
> I'd like to get some feedback regarding the following
> approach to supporting multicast groups in userspace, and in
> particular for MPI.  Based on side conversations, I need to
> know if this approach would meet the needs of MPI developers.
> 
> To join / leave a multicast group, my proposal is to add the
> following APIs to the rdma_cm.  (Note I haven't implemented
> this yet, so I'm just assuming that it's possible at this point.)
> 
> /* Asynchronously join a multicast group. */ int
> rdma_set_option(struct rdma_cm_id *id, int level, int optname,
> 			  void *optval, size_t optlen);
> 
> /* Retrieve multicast group information - not usually called.
> */ int rdma_get_option(struct rdma_cm_id *id, int level, int optname,
> 			  void *optval, size_t optlen);
> 
> /*
>  * Post a message on the QP associated with the cm_id for the
>  * specified multicast address.
> */
> int rdma_sendto(struct rdma_cm_id *id, struct ibv_send_wr *send_wr,
> struct sockaddr *to); 
> 

If we are going to add mullticasat group logic to rdma_cm then
we probably should have a multicast definition that encompasses
UDP as well as IB UD.

For the most part, implementing a "UD QP" over UDP in pure
software would be fairly simple. The key issue is that
most RNICs do not have stateful offload for UDP, and
therefore the work completions for the "UD QP" would
be generated on the host rather than on the RNIC.

I have a question for applications that want to use
mullticast and point-to-point. Is it acceptable to
require that UD QPs feed *different* CQs than the
CQs fed by RC QPs? Such a restriction would allow
software "UD QPs" to feed software "UD CQs" without
having to complicate the existing hardware CQs.

With that issue solved, the remaining issues look
very mappable.