[openib-general] RE: [RFC] DAT 2.0 immediate data proposal

[Kanevsky, Arkady]

comments on Arlin and Caitlin's emails inline.

Arkady Kanevsky                       email: arkady at netapp.com
Network Appliance Inc.               phone: 781-768-5395
1601 Trapelo Rd. - Suite 16.        Fax: 781-895-1195
Waltham, MA 02451                   central phone: 781-768-5300
 

> -----Original Message-----
> From: Caitlin Bestler [mailto:caitlinb at broadcom.com] 
> Sent: Monday, January 30, 2006 7:16 PM
> To: Arlin Davis; Kanevsky, Arkady
> Cc: Lentini, James; dat-discussions at yahoogroups.com; 
> openib-general at openib.org
> Subject: RE: [openib-general] RE: [RFC] DAT 2.0 immediate 
> data proposal
> 
> Arlin Davis wrote:
> > Kanevsky, Arkady wrote:
> > 
> >> Arlin,
> >> I am not convinced we need a new recv for immediate data.
> >> But what is needed is change in normative text in many places.
> >> Recv, RDMA Write, DTO completion events, error behavior.
> >> Sure you can define immed data in extension but it still effects 
> >> behavior of the normative part of the spec.
> >> 
> >> 
> > How does it effect the normative part of the spec outside 
> of the DTO 
> > event extension? The post_recv behaves exactly the same.

We will need a paragraph that size of the recv buffer
shall accommodate immediate data if the recv may be matched
with rdma_write_immed. There we can reference Provider attribute
for how immed data is returned. Then in Advice to Consumer
state how to generate transport independent recv
and how it can be optimized based on Provider attr.

> > 
> >> This is why my preference is to put it into the main spec.
> >> 
> >> 
> > ok, with no new recv_immed call we do get a little closer.
> > 
> >> The xfer_size is minor thing. We just need to define it 
> meaning with 
> >> respect to immed_data. Defining it either way is fine.
> >> 
> >> Handling extra space on CQ can be handled by Provider.
> >> We can add a new EVD attribute for the use for handling RDMA_write 
> >> with immed data and Provider can automatically add extra 
> space on CQ.
> >> Provider is already responsible to handing user a single 
> completion.
> >> SO it will only be used for error handling.
> >> 
> >> 
> > sounds good.
> > 
> >> Error handling takes maost of the new write up anyhow.
> >> Regardless where it is done in the spec or in extension.
> >> 
> >> Question on do we want to support Send with immed_data have to be 
> >> decided. Ditto remote RMR invalidation with new post(s) for 
> >> immed_data.
> >> Just because IB supports all possible correlation under 
> one Send post 
> >> does not mean that uDAPL should follow that too.
> >> 
> >> 
> > I would agree, strike them all except rdma_write_immed.

The only one which need to be discussed it Remote invalidate
with rdma_write_immed and Local invalidate with rdma_write_immed.

> > 
> > Can you give some idea how you would write up the normative 
> text for 
> > the transport independent receive that would accept immediate data?
> > 
> > thanks,
> > 
> > -arlin
> 
> The data source:
> 	posts an rdma write with immediate DTO, supplying
> 	the RDMA Write data source and an immediate value.
> 
> 	This is translated into one work request (if the
> 	device supports write with immediate), or into
> 	a RDMA Write followed by a RDMA Send (if it does
> 	not). 

This should be Model Implication section.

> 
> 	While successful completion of the RDMA Write will
> 	be suppressed, the Consumer must still allow for the
> 	extra space on the SendQ and the CQ. An IA attribute
> 	will document how many work requests a write_with_immediate
> 	will translate into.

This belongs to Model implication also and in Usage section.

> 
> The data sink:
> 	post a recv (to EP or SRQ) with a four byte buffer.
> 
> 	When it reaps the completion it needs to be ready
> 	to see the data either in an "immediate" field in
> 	the work completion, or in the buffer originally
> 	specified in the recv DTO.
> 

This is in the Usage section.

> 
> A Provider MAY indicate that it supports immediate receives, 
> but on iWARP or any transport where this is not the default 
> optimized receive processing MUST be enabled by the user.
> Otherwise, RFC compliance would require that a four byte 
> untagged message matched to a zero byte buffer was an error.
> Essentially the user is posting a receive operation that 
> names the four bytes in the Work completion as the buffer.
> 

Ditto. Also it should reference Provider Attribute and not transport.

>