[openib-general] RDMA CM and loopback addresses

[Sean Hefty]

>Well, if you have to know the HW devices at bind() rather than at
>connect()/accept() (in socket speak) then your right, I don't think
>you can ever match the full semantics for IP routing in linux :< This
>maybe isn't so bad for IB (only 1 IP subnet), but it seems regrettable
>for iwarp.

On the server side, the server doesn't _need_ to know which RDMA device to use
until the connection request is received.  I.e. bind() doesn't need to specify
an IP address.  The current behavior is that if an IP address is specified, then
that IP address is used to map to a specific RDMA device, to allow allocating
resources before a connection request is received.

On the client side, the client does need to know which device to use before
calling connect.  The client obtains the device by calling bind() or
resolve_addr().  The latter call takes the source and destination addresses, and
obtains the device based on local routing information.

So, bind() could be made to avoid mapping to a specific hardware device.  I just
don't think that it makes as much sense for RDMA interface, and results in other
restrictions, such as requiring deferred resource allocation.

>Does that also mean if you choose to use a wildcard listen address
>then you cannot pre-allocate QPs/etc? Or are true wildcard RDMA
>listen connections just not supported?

True wildcard listens are supported.  The RDMA CM simply initiates a listen on
every RDMA device in the system.  However, if a wildcard listen is used, the
user cannot pre-allocate QPs/etc.

This assumes that the user is letting the RDMA CM handle device hot plug events
for them.  A user that registers for and handles device add/remove events
themselves wouldn't have this restriction.

>This is inevitable, but all the route selection stuff can apply to the
>client side to choose the local device and on the server to vett that
>an incoming connection is administratively valid.

On the client side, route selection is used where possible.  Based on your
previous mail, there may be an issue with how the local device is located, but I
don't think it's anything that couldn't be fixed.  On the server side, no
validation is done, so that would need to be added.

- Sean