[ofiwg] completion flags as actually defined by OFI

[Jason Gunthorpe]

On Wed, Apr 15, 2015 at 12:07:21AM +0000, Hefty, Sean wrote:
> > *FI_DELIVERY_COMPLETE*
> > : For reliable:
> > 
> >   Indicates that a completion should be generated when the work
> >   request is delivered to the peer. FI_DELIVERY_COMPLETE guarentees
> 
> I don't like this wording.  For an operation such as RMA read or an
> atomic, the completion isn't related to the delivery of the work
> request, but that data has arriving at the initiator.

Yes, this wording clearly only applies to local work requests that
create a peer completion. So the SEND family and RDMA WRITE.

It doesn't make sense to apply any of this discussion to work that
doesn't have a peer completion. There is only one possible completion
model for READ and ATOMICS:

RDMA READ: local completion means the data is read from the peer and
available in the local buffer. There is no peer completion.

ATOMICS: local completion means the atomic operation is completed at
the peer, the peer's CPU and all other nodes view the new value, and
the new value is present in local memory. There is no peer completion.

Even the 'no completion' model doesn't make much sense for READ and ATOMIC.

[Granted there may be completion model if you involve persistent NVM, but
 again, without HCA hardware support such a thing doesn't exist today]

So my assumption/intention was that this entire text was qualified to
apply to SEND/WRITE.

> > *FI_COMMIT_COMPLETE*
> > : Indicates that a completion should not be generated until the
> >   completion has been delivered to the peer, consumed by the
> >   application and acknowledged to be complete.
> >   [this needs more language, what api does the application use to
> >    signal it completed the work?]
 
> This seems to get into an application level ACK, versus describing
> the delivery of data into memory.

Yes, see my other comments for detail.

Jason