[ofiwg] completion flags as actually defined by OFI

[Hefty, Sean]

> > *FI_INJECT_COMPLETE*
> > *FI_TRANSMIT_COMPLETE*
> 
> From an application perspective, thinking about reliable, I can't tell
> what the difference is between FI_INJECT_COMPLETE and
> FI_TRANSMIT_COMPLETE.
> 
> Neither seem to guarentee a completion on the peer? Neither seem to
> synchronize with shutdown? Why should an application care?

Shutdown / finalize must still be handled by the application.  INJECT is only useful if the app needs to reuse its buffer as quickly as possible, and doesn't want to wait for timeout/retries.

> The shutdown race still exists, one option makes it smaller?

The transmit option at least tells the app that the data has been sent and any retries is outside the responsibility of the local provider.  With inject, the data may still be in local data buffers.  Based on my testing, the difference in the data arriving at the destination ends up being an order of magnitude better with transmit versus inject (for a provider that distinguishes between the two models).