[ofiwg] A question on FI_DELIVERY_COMPLETE

[Hefty, Sean]

> Here’s my understanding of how FI_DELIVERY_COMPLETE works on the
> *responder* end:  If you are doing an RMA operation, and the requester
> uses CQ_REMOTE_DATA to signal the end of the transfer to the responder,
> and the responder has FI_DELIVERY_COMPLETE set, then the responder won’t
> get a completion event until the data is actually visible to the
> responder.
>
> I ask because the man pages imply that FI_DELIVERY_COMPLETE, which is an
> operation flag, applies only to the requester side.  But it is much less
> important to notify the requester that data is visible to the responder,
> than it is to notify the responder itself.

FI_DELIVERY_COMPLETE is intended only to apply to the initiator of an operation.

The generation of a notification at the target is assumed to occur after the operation has completed -- i.e. any transferred data is available.  This holds whether the completion is an entry placed into a CQ, or a completion counter has been incremented.  The FI_REMOTE_CQ_DATA flag is somewhat independent of this.  That flag just means that application data was written into a CQ entry.

Although IB cannot generate target notification without FI_REMOTE_CQ_DATA (i.e. immediate data), libfabric does not require this.

The generation of a completion entry at the target is independent of the completion mode selected by the initiator.

- Sean