[openib-general] multicast

[Sean Hefty]

>Yes - I'm actually talking about a separate issue here.  It looks like
>using the RDMA CM for multicast is going to require using it for all of
>my connection management, so I'm looking at what that entails.
>Currently I'm using only ibverbs and Open MPI's runtime environment layer.

The RDMA CM is designed so that you can handle RC, UD, and multicast
communication through it using a single addressing scheme (IP).  It lets you mix
UD and multicast traffic on the same QP.

>That would be very nice - Open MPI already supports enumeration of IP
>interfaces (which I could do rdma_bind_addr() on as you suggested) in a
>portable fashion, but I think being able to get this via RDMA CM is a
>better general solution.

I see two reasonable solutions here.  One is to provide an enumeration of RDMA
IP addresses.  The second is to use the existing methods to enumerate all IP
addresses, then provide a check to determine if an IP address is usable as an
RDMA address.  The current code does the latter; although, not necessarily in an
intuitive way.

>I understand RDMA CM is a generic CM intended for other types of devices
>(ie iWARP), not just infiniband.  Will all of these devices be supported
>under the ibverbs interface?  I'm thinking it would be a problem if
>we're picking up interfaces that don't support ibverbs, then try to use
>ibverbs to communicate over them.

The intent is that, yes, all RDMA devices should fit under the ibverbs (to be
renamed rdma_verbs) interface.  Changes would be made to ibverbs to accommodate
new devices as they come along.

>Sorry - I'm talking about IP addresses of the local ipoib devices, or
>whatever sort of addressing structure a particular network uses.  Yes,
>we currently send this information out of band, over TCP.

If you're using IP addresses for all RDMA devices, do you still need to exchange
this information (which is also going over an IP address)?  Is this to get more
control over the routes used for connections?  Do you do this even when using
standard IP over Ethernet?

>At the moment, I'm more concerned about how the RDMA CM API fits with
>Open MPI (which I think it will, just need to re-think connection
>management).  In the future though, one thing that comes to mind is
>control of dynamic/multipath routing.

You have greater control over paths by using rdma_get_option / rdma_set_option.
There are options available to retrieve a list of all possible path records from
the local node to a specified destination.  You can then manually set which path
you want a connection to use.  Use of these options require that the local SA
cache be enabled.

- Sean