[openib-general] Re: [Rdma-developers] Meeting (07/22) summary:OpenRDMA community development discussion

[Caitlin Bestler]

> -----Original Message-----
> From: rdma-developers-admin at lists.sourceforge.net 
> [mailto:rdma-developers-admin at lists.sourceforge.net] On 
> Behalf Of Yaron Haviv
> Sent: Sunday, July 31, 2005 8:57 PM
> To: Christoph Hellwig; Tom Duffy
> Cc: Venkata Jagana; rdma-developers at lists.sourceforge.net; 
> openib-general at openib.org
> Subject: RE: [openib-general] Re: [Rdma-developers] Meeting 
> (07/22) summary:OpenRDMA community development discussion
> 
> > -----Original Message-----
> > From: openib-general-bounces at openib.org [mailto:openib-general- 
> > bounces at openib.org] On Behalf Of Christoph Hellwig
> > Sent: Friday, July 29, 2005 8:02 AM
> > To: Tom Duffy
> > Cc: Venkata Jagana; rdma-developers at lists.sourceforge.net; 
> Christoph 
> > Hellwig; openib-general at openib.org
> > Subject: [openib-general] Re: [Rdma-developers] Meeting (07/22) 
> > summary:OpenRDMA community development discussion
> > 
> > On Thu, Jul 28, 2005 at 02:02:08PM -0700, Tom Duffy wrote:
> > > At OLS (and in previous forums), the kernel maintainers 
> have made it
> > > *very* clear that there should only be one API.
> > 
> > _and_ that this api is neither RNIC-PI or KDAPL.  In fact 
> for anything 
> > that doesn't look very similar to the current IB midlayer 
> you'd need 
> > very convincing arguments.
> > 
> 
> I assume it is not as simplistic as that iWarp CM model is 
> quite different than IB, and iWarp doesn't have SA/SM and a 
> bunch of other IB specific things 
> 
> For example: 
> The correct common abstraction is one where a user can issue 
> a connection by using a logical end-point address (such as an 
> IP), and doesn't have to deal with the IB or iWarp specific 
> CM state machine or SA/SM. 
> 
> If you look at DAPL you can break it to simple Verbs (e.g. 
> send, ..) where its just a simple overlay on to of the verbs 
> (and may be
> redundant)
> However there is a second part that implements a simple 
> connection establishment model (much like BSD) that can be 
> mapped to both IB (CM, SA, ..) or iWarp (TCP Syn/SynAck, ARP, 
> etc'), this serves couple of main
> purposes:
> a. make it simple for ULP developer and put the complex part 
> in a common
> place   
> b. define a common model for different HW
> 
> we can spend time and discuss theories and intentions, at the 
> end of the day an iWarp RNIC cannot just reside under 
> IB-Verbs without major changes to the overall infrastructure.
> Several guys spent some time looking it over and came with an 
> abstraction that IS possible on top of IB & iWarp & foo, that 
> is called DAPL (or IT as another similar alternative)
> 
> It would probably be wise to try and merge that effort with 
> IB-verbs etc' 
> (e.g. make the verbs portion of the API closer), and on the 
> same time preserve the effort that was done in kDAPL to 
> overcome the differences (e.g. in the CM, addressing portions)
> 

The working theory is that two additional connection management
'verbs' will be proposed, both optional.

The first is a straight-forward mapping of the traditional DAT
connection establishment: i.e., do whatever you have to do in
order to listen, request a connection, accept/reject connection
requests. This is an interface that can map to existing iWARP
implementations very easily while not immediately standardizing
integration with the host TCP stack.

Connection Requests are reported in this model, but via callbacks
rather than EVDs, since it is desired to keep a verb layer interface
more spartan than kDAPL. However, the fact that connection requests
must be approved allows blocking of all requests that conflict
with host policy (as expressed in packet filtering rules).

It is also an interface that IB vendors *could* support, because
DAPL implements over IB. We could consider having a default
implementation of the methods from the DAPL code for that
purpose.

After that we need a standardized way to implement "modify qp
to RTS" in a iWARP-centric fashion. This will require agreement
on how to transfer the TCP state from the host stack to the
RNIC. This is the more flexible model that matches RDMAC
verbs and supports all connection establishment models, 
including iSER. It is inherently iWARP specific, however,
since IB connections do not start their life as TCP
connections. 

The benefit of this second additional API is that the 
preserves *all* host stack safeguards on connection 
establishment because the host stack actually establishes
the TCP connection. It does require agreement on the correct
way to extract the TCP connection from the host stack,
however. This is something that existing deployments
already do, but they aren't doing it in mainline code.
Reaching consensus on a sustainable interface for doing
this is certainly worthy of careful consideration, which
is why the first additional DAT-style API will be proposed.