[openib-general] RDMA connection and address translation API

[Fab Tillier]

> From: James Lentini [mailto:jlentini at netapp.com]
> Sent: Wednesday, August 24, 2005 1:58 PM
> 
> On Wed, 24 Aug 2005, Fab Tillier wrote:
> 
> > > From: Roland Dreier [mailto:rolandd at cisco.com]
> > > Sent: Wednesday, August 24, 2005 11:03 AM
> > >
> > >     Fab> Why can't the IPV field be ignored?  If a listen wants only
> > >     Fab> IPV4 addresses, it would specify a 16-byte compare buffer
> > >     Fab> with the first 12 bytes zero, the next 4 filled with the IPV4
> > >     Fab> address, and would set the offset to that of the hello
> > >     Fab> message's destination address (32).
> > >
> > > Yes, you're right for SDP.  I guess if we're comfortable mandating
> > > that all protocols put their source and destination IPs in the private
> > > data for the IB case, then this works.  Of course it's somewhat
> > > awkward to pass this information into the transport-neutral CM API but
> > > I think this can be worked around.
> >
> > I don't know if we need to mandate IP usage - it's up to the
> > application.  Any application that wants to have similar semantics
> > to the way socket listens work (especially when bound to one of
> > multiple IP addresses on a port) the application would have to
> > define its private data to accommodate this.
> >
> >  At the IB level, the contents of the private data are still opaque,
> > even to the CM.  The CM would only expose the ability to have it
> > perform an initial triage of requests by doing binary comparisons
> > over regions of private data.  It doesn't know (or need to know)
> > what the data represents - it only cares about finding a match (or
> > not).  The CM doesn't define any sort of policy here, and I don't
> > think it should.  It's just bytes to the CM, and it's doing a blind
> > comparison without interpreting the contents.
> 
> You need to consider what makes sense for *both* ib and iwarp. Keep in
> mind that the correct API will allow a consumer to use ib and iwarp
> devices transparently. In other words their will be one code path that
> support both.

I believe using the private data makes the most sense from the IB perspective.
One could even argue that it is the only way to provide positive "getpeername"
functionality.  Use of the IB private data does not require identical use of
private data in other technologies.

> If we were to adopt your proposal, the consumer would need to perform
> unnecessary operations on iWARP.

It doesn't have to impact the client if there's some intermediate abstraction to
isolate the client from the IB CM details (including private data use).

> A transport neutral client would be forced to put IP information into
> its CM private data on iWARP.
> 
> Likewise, a transport neutral server would be forced to pass an
> private data offset and binary blob to the listen API call on iWARP.
> 
> Neither of these make sense.

A higher-level CM abstraction could implement the policy of private data use
when running on IB without the client's involvement.  The end result still is
that you end up with a wire protocol that needs to be documented so that someone
without that exact CM abstraction knows where and how to format the private data
as well as how to interpret it.  If the IBTA defines something like this, all
these issues go away.  I don't know if the IBTA can define this without
affecting existing protocols like SDP and iSER that already define how to
encapsulate the source and destination information in the private data.

Using the private data, either by the client or some IB-specific CM abstraction,
will remove the need for any reverse lookups.  A forward lookup to validate the
incoming source GID to the source IP in the private data can validate the IP
address.  Performing a forward lookup via ARP is going to be a lot faster than
ATS if the ARP entry already exists.  On large fabrics, ARP is also going to
scale better since there's not one single entity responsible for responding to
every node's requests. 

> These API problems are secondary to the burden you would be placing on
> the protocols. As has been mentioned in a previous email, extending
> the current protocols to use this convention will require further
> standardization and in some cases may not be compatible with their
> current architecture.

I think biting the bullet now on establishing these standards for applications
using IP addressing over IB, whether in the IBTA or in each application, is
going to give us the best long term result.

- Fab