[libfabric-users] Two-stage completion

[Hefty, Sean]

> ...oh? I thought that FI_TRANSMIT_COMPLETE was the local completion and
> FI_DELIVERY_COMPLETE was the remote completion. What does this mean,
> then?
> 
> FI_TRANSMIT_COMPLETE
> 	Applies to fi_sendmsg. Indicates that a completion should not be
> generated until the operation has been successfully transmitted and is
> no longer being tracked by the provider.
> FI_DELIVERY_COMPLETE
> 	Applies to fi_sendmsg. Indicates that a completion should be
> generated when the operation has been processed by the destination.

You looking at the flags discussion for the send/receive operations.  This is calling out that those flags only apply to the fi_sendmsg call.  Other send operations do no take flags, and they do not apply to the fi_recvmsg call.

These flags can set the completion model for a specific send operation.  The documentation here does not try to re-state the full meaning of the completion mode, however.

> Anyway, I realize that I'm trying to have my cake and eat it too, but
> in general, I'm looking for:
> 	1. Blocking send semantics over unconnected endpoints

Libfabric does not define blocking operations.  All operations are asynchronous.

> 	2. You get to send again as soon as the buffer is safe to write
> to (currently my use case for the cq),

The buffer may be re-used either after you get a completion or immediately if an inject call is used.

> 	3. You also get some kind of event when we're sure the
> destination received the event,

You need to clarify what it means for the destination to receive the message.  Is the destination the peer process?  Peer node?  Peer NIC?

> 	4. The application doesn't perform extra copy operations on the
> message unless it's completely unavoidable.

Well, what exactly is an 'extra copy'?  :)

The API does not dictate how a provider implements the various completion semantics.  Providers may copy the message buffers into outbound message queues, some internal buffer, or whatnot.  You just hope that the provider selects the best performing option.

> This doesn't mean "return from the send call with the buffer safe
> ASAP," as in that case I would just use the memcpy strategy.
> 
> So is fi_tinject with FI_INJECT_COMPLETE what I want? It seems that
> that's probably the case, but I do need unconnected endpoints. Since
> the man pages say that it's an "optimized version of fi_tsend" it leads
> me to believe I cannot use it without establishing a connection.

The inject and tagged message calls will work with unconnected (RDM - reliable datagram) endpoints.