[openib-general] RE: [dat-discussions] socket based connectionmodel for IB proposal - round 3

[Tom Tucker]

On Tue, 2005-11-15 at 10:33 -0800, Caitlin Bestler wrote:
> openib-general-bounces at openib.org wrote:
> > Kanevsky, Arkady wrote:
> >> Which entity is responsible to "use" the proposed protocol is an
> >> interesting one. I was assuming that this will be CM. After all the
> >> proposed protocol is CM extension protocol. But it can be another
> >> entity module between CM and ULP. 
> > 
> > The use of a reserved bit in the CM message indicates that
> > the CM itself needs to set this data.  This requires
> > communication between the CM and IPoIB - in essence making
> > IPoIB part of the CM.  Removal of the reserved bit is what
> > permits another entity between the CM and the ULP to perform
> > this task.
> > 
> 
> The use of a reserved bit allows the receiver to know that the
> IP address has some validity without having to do a translation
> of the IP address itself.
> 
> As Fab pointed out, if the receiving software takes this step
> then there is no need for any additional CM bit. Because the
> receiver can rely on the translation being authentic. It's
> an extra round trip, but it does leave the local CM interface
> totally intact.
> 
> > 
> >> DAPL APIs (uDAPL and kDAPL) does not expose local IP address for
> >> listen point. An additional API can be added to support passing
> >> local socket to listen on instead of Connection Qualifier. Since it
> >> is addition no backwards compatibility issues.
> >> The current ULPs/Apps will still use "the default API address" and
> >> the protocol assigned SID as connection qualifier.
> > 
> > The issue with DAPL is that it assumes that addressing has
> > been resolved to a specific device before communication
> > between the client and server have even occurred.  I.e. the
> > server must be clairvoyant and know which device a connection
> > request will be received on.  Likewise, a client must assume
> > which local device is needed to connect to a given remote address.
> > 
> > - Sean
> 
> That is actually inherent in any RDMA model. You have to pre-post
> receive buffers before you connect. Therefore you need to know 
> which device to register memory to. You don't need to know the
> external physical port, but you do need to know the device that
> is responsible for memory registration.

Er, I think the current CMA implementation is a valid "RDMA model" and
specifically doesn't require that you determine the local device before
connecting.

BTW, assuming the semantics specified in RNIC Verbs, you don't have to
post recv buffers before connect because the active side must send the
first RDMA message. Therefore, if the passive side posts recv buffers
prior to completing the LLP transition to RDMA mode and the active side
posts recv buffers prior to sending the first RDMA message, there is no
timing hole where a valid RDMA message can be received prior to having a
posted recv buffer to handle it.

> 
> And this is not require clairvoyance. It requires integration
> with the host local routing tables. Something that would be
> easy if the GID were treated as an IPv6 address. But even with
> some form of translation it is easy, as long as the IP addresses
> are integrated with the local routing tables.

The current CMA does this.
> 
> Given a remote IP address (or a local one that you want to use)
> you know what egress port will be used (and which ones could be
> used), and you know that RDMA device(s) associated with those
> egress points. The last step is simple, but has been overlooked
> all too often.

The CMA does this too.
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general