[ofiwg] [libfabric-users] Two-stage completion

[Hefty, Sean]

> What if your provider copies to local buffer, then transmits from
> there? Local completion happens at the rate of memcpy, which is usually
> faster than RDMA.

FI_TRANSMIT_COMPLETE is not a local completion.

"A completion guarantees that the operation is no longer dependent on the fabric or local resources."

I think you're thinking of FI_INJECT_COMPLETE and the fi_inject call, which doesn't return until the operation has completed locally.

FI_DELIVERY_COMPLETE adds to transmit complete that the result is visible to the application.

Transmit complete likely occurs after the remote NIC acks the message, but before the data may have been placed in memory.  Delivery complete likely means that the data is in memory.