[openib-general] RE: [dat-discussions] [RFC][PATCH] OpenIB uDAPL extension proposal - sample immed data and atomic api's

[Caitlin Bestler]

dat-discussions at yahoogroups.com wrote:
> James and Arkady,
> 
> 
> 
> DAPL provides a generalized abstraction to RDMA capable
> transports. As a generalized abstraction, it cannot exploit
> the unique properties that many of the underlying
> platforms/interconnects can provide so I would like to
> propose a simple (minimum impact on libdat) extensible
> interface to uDAPL that will allow vendors to expose such
> capabilities. I am looking for feedback, especially from the
> DAT collaborative. I have included both a design document and
> actual working code as a reference.
> 
> 
> 
> The patch provides a fully tested DAT and DAPL library
> (openib_cma) set with the following provider extensions:
> 
> 
> 
> DAT_RETURN
> dat_ep_post_write_immed(
>       IN DAT_EP_HANDLE        ep_handle,
> 
> IN DAT_COUNT            num_segments
> IN DAT_LMR_TRIPLET      *local_iov,
> IN DAT_DTO_COOKIE       user_cookie,
> 
> IN DAT_RMR_TRIPLE       *remote_iov,
> IN DAT_UINT32           immediate_data,
> IN DAT_COMPLETION_FLAGS completion_flags);
> 
> 

There was an earlier discussion on "extended RDMA Writes" that
could optionally include immediate data and/or rdma-peer-confirmed
delivery. I think any discussions regarding RDMA Writes should
build upon that, since we already outlined the differences 
between iWARP and IB and how each could be supported.

To recap: it is easy to support any of the IB RDMA Write
semantics over iWARP, but doing so maps to a variable number
of work requests (for example, an RDMA Write *and* an RDMA
Send). As I recall, there was insufficient interest to deal
justify the extra complexity. But there is a base of interest
in being able to do these "extended RDMA Writes" in a transport
neutral fashion.

> 
> DAT_RETURN
> dat_ep_post_cmp_and_swap(
>       IN DAT_EP_HANDLE        ep_handle,
> 
> IN DAT_UINT64           cmp_value,
> IN DAT_UINT64           swap_value,
> 
> IN DAT_LMR_TRIPLE       *local_iov,
> IN DAT_DTO_COOKIE       user_cookie,
> 
> IN DAT_RMR_TRIPLE       *remote_iov,
> IN DAT_COMPLETION_FLAGS completion_flags);
> 
> 
> 
> DAT_RETURN
> dat_ep_post_fetch_and_add(
>       IN DAT_EP_HANDLE        ep_handle,
> 
> IN DAT_UINT64           add_value,
> 
> IN DAT_LMR_TRIPLE       *local_iov,
> IN DAT_DTO_COOKIE       user_cookie,
> 
> IN DAT_RMR_TRIPLE       *remote_iov,
> IN DAT_COMPLETION_FLAGS completion_flags);
>

IT-API already has made a similar proposal, and the idea
that atomic work requests at least exist is present in
all of the verbs layer APIs (with the caveat that the
response might be "illegal request").

It might be interesting to explore if there is any
way that these can be defined so that they could easily
be implemented at the driver or Provider layer since
they are obviously not a part of an iWARP RNIC.

I belive this is possible if there is no other
untagged traffic (send/recv) on the same connection.

Would transport neutral atomics with that restriction
be of any use at the application layer? Or is just being
told "no we don't support hardware-assisted atomics"
a simpler attribute?

Keep in mind that atomics were not omitted from iWARP
by oversight. They would have been a fairly simple opcode
to add to the same work queue that handles RDMA Read requests.
It was the consensus of the drafters that IB atomics had not
justified the silicon they required.

The key question is whether the benefits of using atomics
when they are hardware optimized can be retained when the
overhead of checking to see if they can be used is added.