[openib-general] RE: [RFC] DAT 2.0 immediate data proposal

[Caitlin Bestler]

Arlin Davis wrote:
> Kanevsky, Arkady wrote:
> 
>> Arlin,
>> I am not convinced we need a new recv for immediate data.
>> But what is needed is change in normative text in many places.
>> Recv, RDMA Write, DTO completion events, error behavior.
>> Sure you can define immed data in extension but it still effects
>> behavior of the normative part of the spec.
>> 
>> 
> How does it effect the normative part of the spec outside of the DTO
> event extension? The post_recv behaves exactly the same.
> 
>> This is why my preference is to put it into the main spec.
>> 
>> 
> ok, with no new recv_immed call we do get a little closer.
> 
>> The xfer_size is minor thing. We just need to define it meaning with
>> respect to immed_data. Defining it either way is fine.
>> 
>> Handling extra space on CQ can be handled by Provider.
>> We can add a new EVD attribute for the use for handling RDMA_write
>> with immed data and Provider can automatically add extra space on CQ.
>> Provider is already responsible to handing user a single completion.
>> SO it will only be used for error handling.
>> 
>> 
> sounds good.
> 
>> Error handling takes maost of the new write up anyhow.
>> Regardless where it is done in the spec or in extension.
>> 
>> Question on do we want to support Send with immed_data have to be
>> decided. Ditto remote RMR invalidation with new post(s) for
>> immed_data. 
>> Just because IB supports all possible correlation under one Send post
>> does not mean that uDAPL should follow that too.
>> 
>> 
> I would agree, strike them all except rdma_write_immed.
> 
> Can you give some idea how you would write up the normative
> text for the transport independent receive that would accept
> immediate data? 
> 
> thanks,
> 
> -arlin

The data source:
	posts an rdma write with immediate DTO, supplying
	the RDMA Write data source and an immediate value.

	This is translated into one work request (if the
	device supports write with immediate), or into
	a RDMA Write followed by a RDMA Send (if it does
	not). 

	While successful completion of the RDMA Write will
	be suppressed, the Consumer must still allow for the
	extra space on the SendQ and the CQ. An IA attribute
	will document how many work requests a write_with_immediate
	will translate into.

The data sink:
	post a recv (to EP or SRQ) with a four byte buffer.

	When it reaps the completion it needs to be ready
	to see the data either in an "immediate" field in
	the work completion, or in the buffer originally
	specified in the recv DTO.


A Provider MAY indicate that it supports immediate receives,
but on iWARP or any transport where this is not the default
optimized receive processing MUST be enabled by the user.
Otherwise, RFC compliance would require that a four byte
untagged message matched to a zero byte buffer was an error.
Essentially the user is posting a receive operation that
names the four bytes in the Work completion as the buffer.