[openib-general] RE: [RFC] DAT 2.0 immediate data proposal

Mon Jan 23 19:39:16 PST 2006

Arlin,
comments inline.

Arkady Kanevsky                       email: arkady at netapp.com

Network Appliance Inc.               phone: 781-768-5395

1601 Trapelo Rd. - Suite 16.        Fax: 781-895-1195

Waltham, MA 02451                   central phone: 781-768-5300

________________________________

	From: Davis, Arlin R [mailto:arlin.r.davis at intel.com] 
	Sent: Monday, January 23, 2006 7:15 PM
	To: Kanevsky, Arkady; Lentini, James
	Cc: openib-general at openib.org; dat-discussions at yahoogroups.com
	Subject: RE: [RFC] DAT 2.0 immediate data proposal

	Arkady,

	Response inline...

________________________________

	From: Kanevsky, Arkady [mailto:Arkady.Kanevsky at netapp.com] 
	Sent: Tuesday, January 17, 2006 7:16 AM
	To: Davis, Arlin R; Lentini, James
	Cc: dat-discussions at yahoogroups.com; openib-general at openib.org
	Subject: RE: [RFC] DAT 2.0 immediate data proposal

	Arlin,

	a few things need to be addressed.

	1. correlation with local and remote invalidate

	This potentially effects both DAT_DTOs and post operations

	How does this differ from normal sends or writes?
	[AK] We had added a new Send_with_Invalidate. The completion
also states

	whether RMR was invalidated and which one. But the text for
interaction

	is added through out the completion and post operations.

	See the latest draft of uDAPL and kDAPL 1.3 specs on the DAT
reflector.

	2. Need a precise defintion for CONFIRM_FLAG definition in a
transport independent fashion.

	What guarantees DAT Provider "provides" on successful local
completion?

	Remote end guarantee?

	My understanding what you are trying to do is create 2 models
one IB and one for iWARP.

	So for IB Consumers will use CONFIRM_FLAG and for iWARP
IMMED_FLAG.

	Provider will indicate in Provider_attr which model it supports.

	The issue I have with it is that I do not see a model that
Consumer can use to create

	a transport independent code.

	It looks like Immed_flag can be made transport independent. But
with "sender" specifying

	the behavior a protocol extension is needed for IB. IB will
always deliver Immediate data

	in the header not a payload and remote Provider can control how
it is delivered to a Consumer.

	But this means that there is no need for DTO_flags for Send
side. Instead it can be

	used for Recv side or controlled purely by Provider.

	Maybe we need to just go back to one model and always deliver
via the event? With the post_recv_immed requirements, other transports
have a mechanism to emulate and create the necessary resources on the
recv side to place idata and copy to event when operation is completed.
Would this work for iWARP?

	Two different models for receiving idata should be avoided if at
all possible.
	[AK] Caitlin already responded to this. 

	3. Need to define error behavior. for new operations, async
errors, EP behavior.

	I will work on updating the draft. post_send_immed will look
much like post_send and post_rdma_write_immed will look a lot like
post_rdma_write with some additional errors based on the post receive
buffer requirement.  
	[AK] Also consider if you want to add remote invalidate to the
new operation. 

	4. Need to define DAT_Provider attributes for immediate data and
dto_flags behavior

	5. Does Solicited_wait completion_flag value now applicable for
RDMA_write for immediate data?

	yes, applicable to  send, send_immed, and write_immed

	6. Is dto_completion_data xfer_length include immediate_data
size or not?

	no
	[AK] It can work both ways. Either we include 4 extra bytes for
immediate data or not.

	Consumer just have to know. The real data always starts at 4
byte boundary into the buffer

	is immediate data is returned inline. We need to state how
immediate data is positioned

	if it is smaller than 4 bytes. 

	7. what memory privilages needed for a recv buffer for immediate
data?

	Based on the operation... write_immed would require write
privileges and send_immed would require recv privileges. 

	8. SRQ interaction?

	Good question. all post_recv_immed or all post_recv?  
	[AK] Will this work for the user model? Not supporting handling
immediate recv and regular recv with potential immediate data on one
SRQ. 

	9. What happens of buffer for recv operation NOT recv_immed is
matched for incomming recv/rdma_write op?

	The rules should be:

	Can receive a send, send_immed, or write_immed with recv_immed. 

	Cannot receive send_immed or write_immed on a recv.

	However, I am not sure how you would enforce this on IB (DTO
error on the receiving side?) since the idata is delivered via CQ and
does not require a special receive post descriptor.      
	[AK] We can make this Provider attribute. Or we can state that
if immed data is return in event

	then there is no error for recv. 

	10. Change dat_ep_post_write_immed to
dat_ep_post_rdma_write_immed to be consistent with current

	terminology.

	Ok

	11. Need to cleanup operation description to make it clear that
Send|RDMA_write and immediate data part

	is a single atomic operation. The current "followed by" language
is misleading.

	Make it explicit that there is a single local DTO completion and
single remote DTO completion.

	Ok, I will clean that up

	12. Is your intension that post_recv_immed can ONLY except
immediate data and is not

	capable to recv any message?

	No, the intention is to extend the post_recv to handle 32bit
idata which may arrive with or without other send or rdma_write data.

	Does it make more sense to add a dto_flags to the existing
post_recv?

	13. size should be num_segments for dat_ep_post_recv_immed()

	ok

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20060123/251d69db/attachment.html>