[ofiwg] completion flags as actually defined by OFI

[Jason Gunthorpe]

On Thu, Apr 16, 2015 at 08:31:25PM +0000, Hefty, Sean wrote:
> These are the new completion flags and their definitions that I have in a pending patch.

The others seem quite reasonable

> *FI_DELIVERY_COMPLETE*
> : Indicates that a completion should not be generated until an operation
>   has been processed by the destination endpoint(s).  

> A completion guarantees that the targets have visibility into the
>  results of the operation.

Suggest:

'the results of the operation are visible to all observers, including
the target's CPU'

However, likely only software based providers can use this model.
Hardware offload HCAs, like mlx, cannot provide this guarentee. PCI-E
is fundamentally incoherent with the CPU until the CPU itself does
some kind of synchronizing operation with the initiator - be it
processing a MSI or reading data from the HCA.

>   This completion mode applies only to reliable endpoints.  For operations
>   that return data to the initiator, such as RMA read or atomic-fetch,
>   the source endpoint is also considered a destination endpoint.  This is the
>   default completion mode for such operations.

IB ATOMIC is not coherent or atomic with the CPU, only to other
atomics from the same card.

So for verbs, it would actually provide FI_TRANSMIT_COMPLETE semantics
for ATOMIC, which seems like nonsense. Maybe call it
FI_DELIVERY_GUARENTEED, for the RC case instead, and have
FI_TRANSMIT_COMPLETE be the label used for the UD case.

And then add this clarification to FI_TRANSMIT_COMPLETE:

 The order of observability is not defined. For instance, the completion
 may be processed at the source before the target CPU can view the
 data.

 Remote atomics are not coherent or atomic with the target CPU.

Jason