[ofiwg] definition of a completion in OFI

[Hefty, Sean]

> In IB terms:
> For reliable services, a WQE is not considered complete until all
> necessary responses have arrived.  (compliance statement C9-60).
> For unreliable services, the requester shall consider a Message Send or
> RDMA WRITE complete when the last byte has been committed to the wire (or
> an unrecoverable error occurs).  (Compliance statement C9-180).
> 
> For bonus credit, can you point to the IB spec location where this is
> clarified?
> [PG] See above.  How shall I collect my bonus credit?

Excellent - thanks!  I'll check the iwarp specs again for similar wording.

I'll save you an extra dessert in Monterrey.

> [PG] I don't think that's a reasonable definition of completion, but the
> point is arguable.   The definitions that IB uses for the requester side
> are the ones given above.  The responder side may be a different story.  I
> think we should adopt the same definitions as used by IB.

IMO, this ends up being too strict, which may limit optimizations.  I can see a use for supporting the existing definition, that simply means that the local buffers can be reused, in order to allow for queueing locally or in the fabric.  A stricter completion model meets this requirement.