[ofiwg] A question on FI_DELIVERY_COMPLETE

[Paul Grun]

-----Original Message-----
From: Hefty, Sean [mailto:sean.hefty at intel.com] 
Sent: Monday, October 26, 2015 12:05 PM
To: Paul Grun; ofiwg at lists.openfabrics.org
Subject: RE: A question on FI_DELIVERY_COMPLETE

> Here's my understanding of how FI_DELIVERY_COMPLETE works on the
> *responder* end:  If you are doing an RMA operation, and the requester 
> uses CQ_REMOTE_DATA to signal the end of the transfer to the 
> responder, and the responder has FI_DELIVERY_COMPLETE set, then the 
> responder won't get a completion event until the data is actually 
> visible to the responder.
>
> I ask because the man pages imply that FI_DELIVERY_COMPLETE, which is 
> an operation flag, applies only to the requester side.  But it is much 
> less important to notify the requester that data is visible to the 
> responder, than it is to notify the responder itself.

FI_DELIVERY_COMPLETE is intended only to apply to the initiator of an operation.
[PG] I suspected as much.

The generation of a notification at the target is assumed to occur after the operation has completed -- i.e. any transferred data is available.  This holds whether the completion is an entry placed into a CQ, or a completion counter has been incremented. 

[PG] Using one of today's well-known networks as an example, there is no way to guarantee that data is actually visible to the responder before posting a completion to the CQ.  I assume that for backward compatibility reasons we would want to maintain that behavior.  That implies that it would be desirable to define some other behavior in which the responder side provider, through some internal mechanism, guarantees that the data is visible to the consumer before signaling the completion to the consumer (whether it is via a completion event or a counter increment).  It would be the moral equivalent of FI_DELIVERY_COMPLETE, but on the responder side. 

 The FI_REMOTE_CQ_DATA flag is somewhat independent of this.  That flag just means that application data was written into a CQ entry.
[PG] I don't quite understand this.  As I read it, remote cq data is the moral equivalent of immediate data, and FI_REMOTE_CQ_DATA is the mechanism that causes the requester to send immediate data.  The presence of this Remote CQ Data, in turn, causes a completion event on the remote side, which might be either an event posted to the completion queue, or the increment of a counter. (In the case of IB, it also causes the consumption of a RECV WQE, but that isn't the case with libfabric.)

Although IB cannot generate target notification without FI_REMOTE_CQ_DATA (i.e. immediate data), libfabric does not require this.
[PG] Other than a subsequent send message, how can libfabric generate a notification on the target side other than using FI_REMOTE_CQ_DATA?

The generation of a completion entry at the target is independent of the completion mode selected by the initiator.
[PG] Agreed.  I am suggesting a mechanism that controls the generation of a completion entry at the responder side.

- Sean
[PG] Attached are a few slides to illustrate.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OFIWG_responder_notification_2015_10_26.pptx
Type: application/vnd.openxmlformats-officedocument.presentationml.presentation
Size: 165025 bytes
Desc: OFIWG_responder_notification_2015_10_26.pptx
URL: <http://lists.openfabrics.org/pipermail/ofiwg/attachments/20151026/c0ea74c7/attachment.pptx>