[openib-general] Re: [RFC][PATCH] OpenIB uDAPL extension proposal - sample immed data and atomic api's

[James Lentini]

On Fri, 23 Dec 2005, Arlin Davis wrote:

arlin> James Lentini wrote:
arlin> 
arlin> > arlin> DAPL provides a generalized abstraction to RDMA capable arlin>
arlin> > transports. As a generalized abstraction, it cannot exploit the arlin>
arlin> > unique properties that many of the underlying arlin>
arlin> > platforms/interconnects can provide so I would like to propose arlin> a
arlin> > simple (minimum impact on libdat) extensible interface to arlin> uDAPL
arlin> > that will allow vendors to expose such capabilities. I am arlin>
arlin> > looking for feedback, especially from the DAT collaborative. I arlin>
arlin> > have included both a design document and actual working code as arlin>
arlin> > a reference.
arlin> > 
arlin> > This is an excellent document and clearly certain applications will
arlin> > benefit greatly from adding this additional functionality.
arlin> > 
arlin> > Since DAPL's inception, the DAT_PROVIDER structure has contained a
arlin> > field called "extension" of type void *. The purpose of this field was
arlin> > to allow for the kind of provider/platform/interconnect specific
arlin> > extensions you describe.
arlin> > 
arlin> > I believe these features can be added without modifications to the
arlin> > current API by defining a particular format for the DAT_PROVIDER's
arlin> > extension data and indicating its presence via a provider attribute.
arlin> > That would require creating an extension document like this one
arlin> > describing an "extension" structure w/ function pointers to the new
arlin> > functions and a well known provider attribute value. 
arlin> > Is there a reason this was not feasible? Would minor modifications to
arlin> > the existing framework be sufficient (perhaps an "extension" event
arlin> > type)?
arlin>
arlin> A single entry point is still there with this patch, I just 
arlin> defined it a little different with a function definition for 
arlin> better DAT API mappings. The idea was to replace the existing 
arlin> pvoid extension definition with this new one. Can you give me 
arlin> an idea of how you would map these extended DAT calls to this 
arlin> pvoid function definition?

For uDAPL, the DAT_PROVIDER structure is defined as follows:

struct dat_provider
{
    const char *                        device_name;
    DAT_PVOID                           extension;
    ...

You could create a well known extensions API by defining a structure 
with several function pointers

struct dat_atomic_extensions
{
    DAT_RETURN (*cmp_and_swap_func)(IN DAT_EP_HANDLE ep_handle,
                                    IN DAT_UINT64 cmp_value,
                                    IN DAT_UINT64 swap_value, 
                                    IN DAT_LMR_TRIPLE *local_iov, 
                                    IN DAT_DTO_COOKIE user_cookie, 
                                    IN DAT_RMR_TRIPLE *remote_iov, 
                                    IN DAT_COMPLETION_FLAGS completion_flags);
    ...
}

and require the dat_provider's extensions member to point to your new 
extension struct.

To make the API easier to use, you could also create macros, similar 
to the standard DAT macros, to reach inside an objects provider 
structure and call the correct extension function.

#define dat_ep_post_cmp_and_swap(ep, cmp, swap, local_iov, cookie, 
remote_iov, flags) \
        (*DAT_HANDLE_TO_EXTENSION (ep)->cmp_and_swap_func) (\
                (ep),               \
                (cmp),              \
                (swap),             \
                (local_iov),        \
                (cookie),           \
                (remote_iov),       \
                (flags))

A drawback to this approach is that adding new extensions requires 
synchronizing with the original extension specification document. To 
eliminate that issue, you could require that the dat_provider's 
extension member point to a typed list of these sorts of extension 
structures.

arlin> What is your opinion on the way I extended event data, dapl 
arlin> event processing, event types, and cookies?

These looked good to me.

I also agree with Arkady that these functions are of such general 
utility that they should be considered for inclusion into the DAT 
specification.