[ofiwg] completion flags as actually defined by OFI

[Hefty, Sean]

> *FI_INJECT_COMPLETE*
> : Indicates that a completion should be generated when the
>   source buffer(s) may be reused.  FI_INJECT_COMPLETE guarentees that
>   the buffers will not be read from again and the application may
>   reclaim them.
> 
>   Any of local failure, fabric failure, or peer local failure can
>   prevent the delivery of the peer's completion.

These change look good.

> 
>   [ Mandatory, all must support this ]
> 
> *FI_DELIVERY_COMPLETE*
> : For reliable:
> 
>   Indicates that a completion should be generated when the work
>   request is delivered to the peer. FI_DELIVERY_COMPLETE guarentees

I don't like this wording.  For an operation such as RMA read or an atomic, the completion isn't related to the delivery of the work request, but that data has arriving at the initiator.  There could also be more complex operations in the future that involve multiple nodes. 

>   that the delivery of the peer's completion is no longer dependent on
>   the fabric or any local resources.

Some operations may require a 3 or 4 way handshake of data.  So, this may not hold true if the completion is associated with the receiving of data at the initiator.

>   A peer local failure can prevent the delivery of the peer's
>   completion.
> 
>   For unreliable:
> 
>   Indicates that a completion should be generated when the work
>   request is delivered to the fabric and is no longer dependent on any
>   local resources. No peer completion is guarenteed.
> 
>   A fabric failure, or peer local failure can prevent the delivery of
>   the peer's completion.

The rest seems reasonable.

> 
>   [IB does this 99% today, presumably sockets/etc are 100%, iWarp does
>    not support this]
> 
> *FI_COMMIT_COMPLETE*
> : Indicates that a completion should not be generated until the
>   completion has been delivered to the peer, consumed by the
>   application and acknowledged to be complete.
>   [this needs more language, what api does the application use to
>    signal it completed the work?]

This seems to get into an application level ACK, versus describing the delivery of data into memory.