[openib-general] RE: [RFC][PATCH] OpenIB uDAPL extension proposal - sample immed data and atomic api's

[Kanevsky, Arkady]

comments inline.

Arkady Kanevsky                       email: arkady at netapp.com
Network Appliance Inc.               phone: 781-768-5395
1601 Trapelo Rd. - Suite 16.        Fax: 781-895-1195
Waltham, MA 02451                   central phone: 781-768-5300
 

> -----Original Message-----
> From: Arlin Davis [mailto:ardavis at ichips.intel.com] 
> Sent: Thursday, January 05, 2006 6:35 PM
> To: Kanevsky, Arkady
> Cc: Arlin Davis; Lentini, James; 
> dat-discussions at yahoogroups.com; openib-general at openib.org
> Subject: Re: [openib-general] RE: [RFC][PATCH] OpenIB uDAPL 
> extension proposal - sample immed data and atomic api's
> 
> Kanevsky, Arkady wrote:
> 
> > Arlin,
> > nice proposal, thanks.
> > I have one high level question and a few specific technical ones.
> >  
> > 1. Why do you want to provide this functionality via 
> extension instead 
> > of part of new DAT spec, say 2.0?
> > This will allow Consumers to use all events, operations, and 
> > Provider/IA functionality uniformly instead of via 2 
> separate layers. 
> > This will also ensure that this basic funcionality can be 
> provided by 
> > all DAPL Provider the same way on DAPL and DAT layers.
> > DAPL 2.0 is not done yet so we have time to incorporate that.
> > DAPL 2.0 already introduced new functionality which is easy 
> to beef up 
> > for your proposal.
> > See DAT_DTOS for example. DAT_EVENT is also modified to 
> handle remote 
> > invalidation so a small addition for Immediate data and 
> Atoimc ops is 
> > a sensible addition.
> > This should simplify proposal significantly. As you will 
> not need to 
> > introduce any new EXT structures.
> 
> As mentioned on the con-call, there are two separate items to 
> consider while looking at the proposal. The first is the 
> ability to extend DAT for specific provider value-add and the 
> second is to validate the need for general atomic and 
> immediate data functionality in the basic set of API's for 
> all providers. I included atomics and immediate data as 
> examples since it is specific to one provider (IB), it 
> includes operations that require new ops, events, and event 
> data types, and it also provides a working model to validate 
> the extension model from request to completion events. I 
> would like to concentrate on getting consensus on the 
> extension proposal first if possible. Just try to think of 
> the actual operations as some opaque dat_ext_foobar_op().

The thing that bothers me is that we already have several APIs
that are transport specific. While some are possible to implement
on other transports the others, like Socket CM, can not.
So I view both of your specific extensions as transport specific
amd hence prefer to add them as normal APIs not extensions.
The secondary goal is that Provider can add extensions without requiring
to change to DAT. These fall into 3 categories.
1. New memory types including privilages and protection attributes.
We can add "extension" entry to these structures. We need to check
if this is sufficient. Think of shared memory for example.
I am assuming no changes to PZ.
2. New DTOs. The main issue is not DTOs but their completions and
async errors. This is why Immediate data is better handled by
incorporating into
DAT spec while atomic can be handled by extensions. That is completion
will return
"extention" and Consumer will do the secondary switch on the extension
type.
Extension should not impact backwards compatibility.
We had not looked at errors. But assuming a simple model that async
errors
break connection and we can return "extension error" with extensions
defining
new reason. Again details need to be polished.
3. new connection types or CM models... New connections seems to have
little impact
on existing API assuming that EP type can be extended. The new
connection can even
restrict which DTO they can handle. CM model is more problematic.

Arlin, it would be nice to consider some of your other extensions that
are not
transport specific to see how it will fit before we make the final
decision.
This should give us idea how extensible DAT "extension" model is.


> 
> >  
> > In general, extension route was intended for RNIC|HCA providers to 
> > expose HW capabilities beyond IBTA, iWARP and VIA standards. The 
> > standard RDMA functionality is best handle via spec addition.
> > DAT 2.0 does it for FMR, remote and local memory 
> invalidation as well 
> > as others.
> 
> True, but the extension route is not fully defined, 
> documented, nor implemented. This is what I would like to 
> work on getting completed in time for 2.0 if possible. 
> 
> BTW: The existing implementation actually uses 
> dapl_provider->extension to store the hca_ptr but the 
> specification states that it is reserved for the providers 
> private use (8.2.1 in DAPL1.2 spec). This is why I had to 
> defined another extension_func in the patch.
> 
> >  
> > I had posted a complete list of changes/addition to DAT 2.0 about a 
> > month ago.
> > But we had not discussed yet version change from 1.3 to 2.0 nor how 
> > much backwards compatibility spec will provide.
> >  
> > 2. What is IMMED_EVENT? is it just immediate data without 
> any payload one?
> > I suggest chnaging the name so it will not use "EVENT". 
> Just call it 
> > NO_PAYLOAD.
> > Do you want to support 2 different way to delivery immediate data?
> > One in event and one in data payload?
> > Why? I would think that just an event way will do.
> 
> This was modeled after the immediate data discussions on the 
> DAT reflector based on iWARP requirements.
> 
> http://groups.yahoo.com/group/dat-discussions/message/3285
> 

I recall it now.
I want to consider a few usage cases.
1. Existing app running on the Provider with extensions.
Want to make sure we do not require any App changes beyond recompile
due to extensions.
2. App wants to be modified to use Immediate data. How big impact it
has on existing code. For example buffer size allocation and completion
handling
for immediate data over existing connection.
2a. Can application take advantage if it knows that Provider will return
immediate data in event?
2b. Immediate data inline only?
3. Ditto for atomic operations over existing connection.

> >  
> > 3. I suggest beefing up DAT_DTO_COMPLETION_EVENT_DATA and 
> DAT_DTOS to 
> > convey which operation completed and return Immediate data 
> if complete 
> > operation had immediate data.
> > Since we already modified these 2 struct as part of DAT 2.0 change 
> > lets add your proposal to the change.
> > This will allow Consumers to use single approach to deal with 
> > completions, extension to the current one but not a 
> structural one. No 
> > need for DAT_EXTENSION_DATA, DAT_EXT_EVENT_TYPE, DAT_EXT_OP nor the 
> > whole mechanism for extended ops.
> 
> You still need extension types for the "other" value-add 
> operations/evnts that will not be accepted as standard and 
> are vendor specific.
>  
> I would like to defer the rest of the questions for now since 
> they touch on actual operations and not the extension 
> mechanism. Although, I do need to think about how to extend 
> memory registration privledges. Any suggestions?

Going with your generic extension design
we add "extension" entry to relevant data structures.
And then outside DAT define the structure for its values which can be
extensible. This imply that adding extension by Provider
will force apps to be recompiled. I hope this is enough.
I am assuming that apps use values not position the structures.

> 
> > 4. What is the purpose of DAT_EXT_WRITE_CONFIRM_FLAG? Is it 
> to expose 
> > IB round trip semantic?
> > iWARP does not support immediate data. One can try to 
> format payload 
> > to pass immediate data.
> > Is that what you had in mind?
> >  
> > What is the semantic meaning of the completion with this flag set? 
> > without flag set?
> > Are extended flags are additonal values for COMPLETION_FLAGS? 2.4.1 
> > talks about extended flags but where they are passed in is not 
> > defined.
> > DAT 2.0 extended them already for FMR barrier. I would prefer to 
> > follow that route rather than creating a separate extension 
> completion 
> > flags.
> >  
> > 5. Why do you need RECV_IMMED? If Immed data is delivered 
> in event no 
> > new Recv operation is needed.
> > If Consumer asks for immediate data in payload where in 
> payload will 
> > it be?
> > If this is needed for local match for remote RDMA_Write to handle 
> > immediate data lets state so.
> >  
> > What happens for mismatch between local and remote op? That is recv 
> > was posted for Send and RDMA_Write "arrived"? Vice Versa?
> >  
> > 6. I see extension for immediate data for rdma_write but 
> not for send. 
> > Is this deliberate? If we are going
> > to extend DAT semantic to support Immediate data we can as well 
> > support the full IBTA/iWARP functionality for it.
> >  
> > 7. Currently memory registration do not support access to 
> LMR or RMR 
> > by Atomic ops.
> > Do you propose to extend the meaning of current MEM_PRIV 
> for LMR and 
> > RMR to covers atomic accesses or add new values to LMR_MEM_PRIV and 
> > RMR_MEM_PRIV for atomic operation support?
> >  
> > 8. Any alignment requirements for memory used for atomic ops?
> >  
> > 9. Any correlation requirements for SRQ buffers to support 
> recv with 
> > immediate data?
> >  
> > Have a great holidays,
> > Arkady
> >  
> >
> > Arkady Kanevsky                       email: arkady at netapp.com 
> > <mailto:arkady at netapp.com>
> >
> > Network Appliance Inc.               phone: 781-768-5395
> >
> > 1601 Trapelo Rd. - Suite 16.        Fax: 781-895-1195
> >
> > Waltham, MA 02451                   central phone: 781-768-5300
> >
> >  
> 
>