[openib-general] RE: [dat-discussions] socket based connectionmodel for IB proposal - round 3

[Caitlin Bestler]

openib-general-bounces at openib.org wrote:
> Kanevsky, Arkady wrote:
>> Which entity is responsible to "use" the proposed protocol is an
>> interesting one. I was assuming that this will be CM. After all the
>> proposed protocol is CM extension protocol. But it can be another
>> entity module between CM and ULP. 
> 
> The use of a reserved bit in the CM message indicates that
> the CM itself needs to set this data.  This requires
> communication between the CM and IPoIB - in essence making
> IPoIB part of the CM.  Removal of the reserved bit is what
> permits another entity between the CM and the ULP to perform
> this task.
> 

The use of a reserved bit allows the receiver to know that the
IP address has some validity without having to do a translation
of the IP address itself.

As Fab pointed out, if the receiving software takes this step
then there is no need for any additional CM bit. Because the
receiver can rely on the translation being authentic. It's
an extra round trip, but it does leave the local CM interface
totally intact.

> 
>> DAPL APIs (uDAPL and kDAPL) does not expose local IP address for
>> listen point. An additional API can be added to support passing
>> local socket to listen on instead of Connection Qualifier. Since it
>> is addition no backwards compatibility issues.
>> The current ULPs/Apps will still use "the default API address" and
>> the protocol assigned SID as connection qualifier.
> 
> The issue with DAPL is that it assumes that addressing has
> been resolved to a specific device before communication
> between the client and server have even occurred.  I.e. the
> server must be clairvoyant and know which device a connection
> request will be received on.  Likewise, a client must assume
> which local device is needed to connect to a given remote address.
> 
> - Sean

That is actually inherent in any RDMA model. You have to pre-post
receive buffers before you connect. Therefore you need to know 
which device to register memory to. You don't need to know the
external physical port, but you do need to know the device that
is responsible for memory registration.

And this is not require clairvoyance. It requires integration
with the host local routing tables. Something that would be
easy if the GID were treated as an IPv6 address. But even with
some form of translation it is easy, as long as the IP addresses
are integrated with the local routing tables.

Given a remote IP address (or a local one that you want to use)
you know what egress port will be used (and which ones could be
used), and you know that RDMA device(s) associated with those
egress points. The last step is simple, but has been overlooked
all too often.