[ofiwg] completion flags as actually defined by OFI

[Jason Gunthorpe]

On Wed, Apr 15, 2015 at 12:35:32AM +0000, Weiny, Ira wrote:

> > To me, the only way the above makes sense, is with dedicated hardware
> > support. Which doesn't exist today. 
> 
> Even then I think the provider can return the FI_COMMIT_COMPLETE
> which indicates an application level of completion ...  Only the
> application knows that the memory is NVM and that storing in NVM is
> what is important.  Libfabric can't make that determination on its
> own.

I could see a future where somone wants to optimize this, it would be
incredibly useful for storage and database scenarios. The goal would
be to have a completion at the sender know that the data is in
peer persistent NVM without involving the peer CPU.

Obviously this requires that the HCA (never mind libfabric) and host
bridge know how to make and handle the right cache bypass and fence
PCI-E TLPs to make this possible.

Without that future, everything involves the CPU, so it may as well
use an application ACK API which allows anything to happen.

Jason