[libfabric-users] question about "default" FI_DELIVERY_COMPLETE semantics for RMA reads and fetching atomics

[Hefty, Sean]

> 
> The fi_cq(3) man page says that
> 
> 	[FI_DELIVERY_COMPLETE] applies only to reliable endpoints. For operations that
> return data to the initiator, such as RMA read or atomic-fetch, the source endpoint is
> also considered a destination endpoint. This is the default completion mode for such
> operations.
> 
> 
> Does "default completion mode" in that last sentence really mean "default", in that
> it's the completion mode that will be used if and only if no
> FI_{INJECT,TRANSMIT,DELIVERY,...}_COMPLETE flag is present in the hints to fi_getinfo()
> and/or the flags to fi_readmsg() etc.?  Or does it really mean that delivery_complete
> is guaranteed for transactions that produce local results, such that even if for
> example FI_TRANSMIT_COMPLETE is given in the hints, RMA reads and fetching atomics will
> nevertheless effectively use delivery_complete?

It's intended to mean the former.  If the application specifically requests some other model, then that is all that it app _should_ assume.  Note that the provider can always provide strong guarantees.

> (I'm pretty sure it's the former, but thought I should check.)
> 
> If it is indeed the former, when an RMA read or fetching atomic is done and
> FI_TRANSMIT_COMPLETE is in force, is there any way to tell when the data has arrived?

This is probably where we can argue that delivery_complete is the minimal semantic that makes sense for RMA read or fetching atomic.

A more technical answer is that a fenced operation can be used to ensure that all prior operations meet the same completion semantic as the fenced op.  So, if a fenced op has deliver_complete, then the completion of the fenced op means that operations prior to that also meet that semantic.

I believe existing hardware provides delivery_complete semantics.  I hesitate to make this a requirement in case someone designs a NIC with more exotic caching capabilities, or if the target memory is non-host memory.

- Sean