[ofiwg] Two-stage completion

[Hefty, Sean]

> In my application, I could benefit greatly if I could force generation
> of two completion events per fi_tsendmsg – one generated on
> FI_TRANSMIT_COMPLETE and the other on FI_DELIVERY_COMPLETE, so I can
> block until the buffer is safe to mangle (post-transmit), and also
> throw an event in my app if a timeout occurs before the delivery
> completion.

Libfabric does not define a mechanism to generate multiple completion events for the same message.  Asynchronous completions are hard enough for most users to deal with.  Making them deal with progress notifications on the way to the operation really completing seems unfriendly.

I would need to understand the benefit here.  The timing difference between the two completion models seems minimal, especially compared to the time needed to generate, transfer, and process an additional ack.

- Sean