[openib-general] kDAPL: ready for gen2/trunk/ inclusion?

[Caitlin Bestler]

On 7/20/05, Libor Michalek <libor at topspin.com> wrote:
> On Thu, Jul 14, 2005 at 02:04:52PM -0400, James Lentini wrote:
> > On Wed, 13 Jul 2005, Christoph Hellwig wrote:
> > > On Wed, Jul 13, 2005 at 10:18:41AM -0400, James Lentini wrote:
> > >> In addition to the streamlined connection management interface, kDAPL
> > >> also provides a unified event model. This is another feature that the
> > >> verbs do not provide.
> > >
> > > It's probably the biggest mis-feature and would have to go aswell if
> > > we mæssage what started as kdapl into the kernel rdma layer.
> >
> > When the DAT Collaborative solicited requirements for their API, RDMA
> > consumers requested this feature. The idea is that a single
> > event source is a cleaner, simpler programming model.
> 
>   I'm not sure how it originally came to be, but it seems like a feature
> that made sense in userspace and got moved to kernel space because of
> the, in my opinion, misguided desire to have a unified kernel/userspace
> API. Having written to the userspace DAPL, and kerel verbs/cm APIs, I would
> not want to use the proposed event model in the kernel. Getting the locking
> correct and good performance is difficult enough without your APIs hiding
> the source of your events.
> 
> 

The fact that it is difficult is exactly why the consumer of kDAPL did
not want to deal with it. They wanted to be able to apply their application
specific logic to an infrastructure that was optimized for the platform.
It seems many of them felt that their application was complex enough
that they preferred not to have to become experts on the OS as well.

While some may consider development of kernel level OS neutral 
applications to be impractical, it is by far from unaminous. Embedded
developers universally prefer this model. It was also the primary mode
of development envisioned by the first DAT developers. The Sourceforge
reference implementation started as a KDAPL implmenetation. One of
the primary motivators in coming up with a new forum to define an RDMA
API was that the existing Kernel VIA interface was subject to proprietary
control. If developers had been confident that they could do everything
they needed in user mode they would have refined VIPL, not created DAT.

It is also important to note that these developers are only asking for
an OS neutral API -- not an OS neutral implementation. It is *their*
code that will deal with OS variants on other fronts, not the KDAPL
code. They only ask that kDAPL exist as a transport *and* OS neutral
API to enable them to concentrate on their application logic.

The evd callback does exactly that. It allows the developer to not
know whether the callback is in an ISR, bottom half or kernel thread.
Instead the application can control the degree, if any, that multiple
callbacks can occur from a single EVD (default is one). This allows
the typical kDAPL application to avoid locks altogether.

Adding a transport neutral CM interface is indeed a required part of
the "verb layer" transport neutral API. And that lower level API should
not deal with things like EVDs. And I still think that there are two
valid approaches to coming up with that lower level API: subset
kDAPL or expand the gen2 verbs.

But both of those will take time. Meanwhile kDAPL as already defined
is available today. It is suitable for almost all kernel RDMA development.
The fact that a more minimalist interface will be refined over the coming
quarters is not a reason that developers should not start using kDAPL
now. As good code engineering we should ensure that kDAPL is built
on top of the evolving verb layer transport. But in the meantime kDAPL
as is already provides transport neutral support. Any lower layer solution
will take time, especially for iWARP vendors.

The goal of encouraging RDMA development is not served by telling 
developers that they'll have to use some variant of kDAPL that is still
under discussion. Telling them that we will support kDAPL (with at
most a few exclusions) and then supplement it with transport neutral
verbs and CM interface for those that need more precise control will
encourage development.