[openib-general] RE: [RFC][PATCH] OpenIB uDAPL extension proposal - sample immed data and atomic api's

[Arlin Davis]

Kanevsky, Arkady wrote:

> Arlin,
> nice proposal, thanks.
> I have one high level question and a few specific technical ones.
>  
> 1. Why do you want to provide this functionality via extension instead 
> of part of new DAT spec, say 2.0?
> This will allow Consumers to use all events, operations, and 
> Provider/IA functionality uniformly instead
> of via 2 separate layers. This will also ensure that this basic 
> funcionality can be provided by all DAPL Provider
> the same way on DAPL and DAT layers.
> DAPL 2.0 is not done yet so we have time to incorporate that.
> DAPL 2.0 already introduced new functionality which is easy to beef up 
> for your proposal.
> See DAT_DTOS for example. DAT_EVENT is also modified to handle remote 
> invalidation so
> a small addition for Immediate data and Atoimc ops is a sensible addition.
> This should simplify proposal significantly. As you will not need to 
> introduce any new
> EXT structures.

As mentioned on the con-call, there are two separate items to consider 
while looking at the proposal. The first is the ability to extend DAT 
for specific provider value-add and the second is to validate the need 
for general atomic and immediate data functionality in the basic set of 
API's for all providers. I included atomics and immediate data as 
examples since it is specific to one provider (IB), it includes 
operations that require new ops, events, and event data types, and it 
also provides a working model to validate the extension model from 
request to completion events. I would like to concentrate on getting 
consensus on the extension proposal first if possible. Just try to think 
of the actual operations as some opaque dat_ext_foobar_op().

>  
> In general, extension route was intended for RNIC|HCA providers to 
> expose HW capabilities beyond
> IBTA, iWARP and VIA standards. The standard RDMA functionality is best 
> handle via spec addition.
> DAT 2.0 does it for FMR, remote and local memory invalidation as well 
> as others.

True, but the extension route is not fully defined, documented, nor 
implemented. This is what I would like to work on getting completed in 
time for 2.0 if possible. 

BTW: The existing implementation actually uses dapl_provider->extension 
to store the hca_ptr but the specification states that it is reserved 
for the providers private use (8.2.1 in DAPL1.2 spec). This is why I had 
to defined another extension_func in the patch.

>  
> I had posted a complete list of changes/addition to DAT 2.0 about a 
> month ago.
> But we had not discussed yet version change from 1.3 to 2.0 nor how 
> much backwards compatibility spec
> will provide.
>  
> 2. What is IMMED_EVENT? is it just immediate data without any payload one?
> I suggest chnaging the name so it will not use "EVENT". Just call it 
> NO_PAYLOAD.
> Do you want to support 2 different way to delivery immediate data?
> One in event and one in data payload?
> Why? I would think that just an event way will do.

This was modeled after the immediate data discussions on the DAT 
reflector based on iWARP requirements.

http://groups.yahoo.com/group/dat-discussions/message/3285

>  
> 3. I suggest beefing up DAT_DTO_COMPLETION_EVENT_DATA and DAT_DTOS
> to convey which operation completed and return Immediate data if 
> complete operation had immediate data.
> Since we already modified these 2 struct as part of DAT 2.0 change 
> lets add your proposal to the change.
> This will allow Consumers to use single approach to deal with 
> completions, extension to the current one
> but not a structural one. No need for DAT_EXTENSION_DATA, 
> DAT_EXT_EVENT_TYPE, DAT_EXT_OP
> nor the whole mechanism for extended ops.

You still need extension types for the "other" value-add 
operations/evnts that will not be accepted as standard and are vendor 
specific.
 
I would like to defer the rest of the questions for now since they touch 
on actual operations and not the extension mechanism. Although, I do 
need to think about how to extend memory registration privledges. Any 
suggestions?

> 4. What is the purpose of DAT_EXT_WRITE_CONFIRM_FLAG? Is it to expose 
> IB round trip semantic?
> iWARP does not support immediate data. One can try to format payload 
> to pass immediate data.
> Is that what you had in mind?
>  
> What is the semantic meaning of the completion with this flag set? 
> without flag set?
> Are extended flags are additonal values for COMPLETION_FLAGS? 2.4.1 
> talks about extended flags
> but where they are passed in is not defined.
> DAT 2.0 extended them already for FMR barrier. I would prefer to 
> follow that route rather than creating a separate
> extension completion flags.
>  
> 5. Why do you need RECV_IMMED? If Immed data is delivered in event no 
> new Recv operation is needed.
> If Consumer asks for immediate data in payload where in payload will 
> it be?
> If this is needed for local match for remote RDMA_Write to handle 
> immediate data lets state so.
>  
> What happens for mismatch between local and remote op? That is recv 
> was posted for Send and RDMA_Write
> "arrived"? Vice Versa?
>  
> 6. I see extension for immediate data for rdma_write but not for send. 
> Is this deliberate? If we are going
> to extend DAT semantic to support Immediate data we can as well 
> support the full IBTA/iWARP functionality for it.
>  
> 7. Currently memory registration do not support access to LMR or RMR 
> by Atomic ops.
> Do you propose to extend the meaning of current MEM_PRIV for LMR and 
> RMR to covers atomic accesses
> or add new values to LMR_MEM_PRIV and RMR_MEM_PRIV for atomic 
> operation support?
>  
> 8. Any alignment requirements for memory used for atomic ops?
>  
> 9. Any correlation requirements for SRQ buffers to support recv with 
> immediate data?
>  
> Have a great holidays,
> Arkady
>  
>
> Arkady Kanevsky                       email: arkady at netapp.com 
> <mailto:arkady at netapp.com>
>
> Network Appliance Inc.               phone: 781-768-5395
>
> 1601 Trapelo Rd. - Suite 16.        Fax: 781-895-1195
>
> Waltham, MA 02451                   central phone: 781-768-5300
>
>