[ofiwg] definition of a completion in OFI

[Jason Gunthorpe]

On Thu, Mar 05, 2015 at 06:37:22PM +0000, Paul Grun wrote:

> In IB terms: For reliable services, a WQE is not considered complete
> until all necessary responses have arrived.  (compliance statement
> C9-60).  For unreliable services, the requester shall consider a
> Message Send or RDMA WRITE complete when the last byte has been
> committed to the wire (or an unrecoverable error occurs).
> (Compliance statement C9-180).

This language is not strong enough to make any meaningful statements
about what happens on the recieve side in the general case. Acks don't
mean the operation will be made visible in the recv q, or in
memory.

If a local side gets an ACK for Send, then immediately tears down the QP
and then immediately fails a incoming packet, the behavior of the
remote side is fairly undefined weather the Send will be in the
recieve q or if the QP error on the send side will eat it. 

AFAIK there is no requirement that the HCA strongly synchronize the
send and recieve sides of a QP when it comes to error cases.

Destroying a RC QP while there is pending send q work on either side
does not have well specified behavior in IB, and that is exactly the case
Sean is probing with his sockets test.

Jason