[ofiwg] completion flags as actually defined by OFI

[Hefty, Sean]

> > *FI_DELIVERY_COMPLETE*
> > : Indicates that a completion should not be generated until an operation
> >   has been processed by the destination endpoint(s).
> 
> > A completion guarantees that the targets have visibility into the
> >  results of the operation.
> 
> Suggest:
> 
> 'the results of the operation are visible to all observers, including
> the target's CPU'

Part of the semantic that needs to be captured is that the target process _may_ need to take additional steps in order to access the data.  For example, the process may need to provide a buffer that the data can be placed into.  I was intentionally vague on the meaning of 'visible', because that is provider/operation/target specific.

> However, likely only software based providers can use this model.
> Hardware offload HCAs, like mlx, cannot provide this guarentee. PCI-E
> is fundamentally incoherent with the CPU until the CPU itself does
> some kind of synchronizing operation with the initiator - be it
> processing a MSI or reading data from the HCA.

OFI tries not to make any assumptions about the implementation.  I can envision ways where hardware offload models can do this, and it may be possible even with current HW through the protocol,  such as following a write by a read.

> >   This completion mode applies only to reliable endpoints.  For
> operations
> >   that return data to the initiator, such as RMA read or atomic-fetch,
> >   the source endpoint is also considered a destination endpoint.  This
> is the
> >   default completion mode for such operations.
> 
> IB ATOMIC is not coherent or atomic with the CPU, only to other
> atomics from the same card.

This isn't attempting to make any claims regarding atomic coherency.  It's attempting to say that the results are visible at both the initiator and target.  For example, the completion of a compare-swap-fetch operation is now accessible by the source and destination.  See my comment above about what it means to be 'visible'. 

- Sean