[openib-general] RDMA CM multicast

[Or Gerlitz]

On 1/24/07, Sean Hefty <sean.hefty at intel.com> wrote:

> What would be needed is a way for the user to indicate that they need a unique
> address.  An obvious way to accomplish this is for the user to specify an IP
> address of 0.0.0.0 when calling rdma_join_multicast().  The user would first
> need to bind to a specific device by calling rdma_bind_addr() with a local IP
> address.

> Your code would look something like this:

> rdma_bind_addr(local IP address)
> rdma_join_multicast(0.0.0.0, port 0)    <- exchange group info out of band
> rdma_join_multicast(0.0.0.0, port 1)    <- exchange group info out of band
> send data to a lot of nodes at once
> rdma_leave_multicast(0.0.0.0, port 0)
> rdma_leave_multicast(0.0.0.0, port 1)

Sean,

This seems to me as a little bit of over engineering... since we do
require that to use the RDMA CM the consumers must have a functional
IPoIB NIC (so they can call rdma_bind_addrress to resolve the
device/port/pkey) we can add another requirement to have the sys admin
configure their routing such that some multicast IP subnet (eg net
224.0.0.0 mask 255.0.0.0) is routed to the IPoIB NIC.

Once this routing is in place, the only thing they need is to enhance
the MPI job starter/etc to allocate to each job (say) two unique
multicast --IP-- addresses on the relevant subnet and provide these IP
addresses to each rank. Now the rank can use the RDMA CM without any
hack.

Or.