[libfabric-users] Two-stage completion

[Smith, Jonathan D]

...oh? I thought that FI_TRANSMIT_COMPLETE was the local completion and FI_DELIVERY_COMPLETE was the remote completion. What does this mean, then?

FI_TRANSMIT_COMPLETE
	Applies to fi_sendmsg. Indicates that a completion should not be generated until the operation has been successfully transmitted and is no longer being tracked by the provider.
FI_DELIVERY_COMPLETE
	Applies to fi_sendmsg. Indicates that a completion should be generated when the operation has been processed by the destination.


Anyway, I realize that I'm trying to have my cake and eat it too, but in general, I'm looking for:
	1. Blocking send semantics over unconnected endpoints
	2. You get to send again as soon as the buffer is safe to write to (currently my use case for the cq),
	3. You also get some kind of event when we're sure the destination received the event,
	4. The application doesn't perform extra copy operations on the message unless it's completely unavoidable.

This doesn't mean "return from the send call with the buffer safe ASAP," as in that case I would just use the memcpy strategy.

So is fi_tinject with FI_INJECT_COMPLETE what I want? It seems that that's probably the case, but I do need unconnected endpoints. Since the man pages say that it's an "optimized version of fi_tsend" it leads me to believe I cannot use it without establishing a connection.

Thanks,
Jonathan


-----Original Message-----
From: Hefty, Sean 
Sent: Wednesday, September 14, 2016 5:14 PM
To: Jeff Hammond <jeff.science at gmail.com>
Cc: Smith, Jonathan D <jonathan.d.smith at intel.com>; libfabric-users at lists.openfabrics.org; ofiwg at lists.openfabrics.org
Subject: RE: [libfabric-users] Two-stage completion

> What if your provider copies to local buffer, then transmits from 
> there? Local completion happens at the rate of memcpy, which is 
> usually faster than RDMA.

FI_TRANSMIT_COMPLETE is not a local completion.

"A completion guarantees that the operation is no longer dependent on the fabric or local resources."

I think you're thinking of FI_INJECT_COMPLETE and the fi_inject call, which doesn't return until the operation has completed locally.

FI_DELIVERY_COMPLETE adds to transmit complete that the result is visible to the application.

Transmit complete likely occurs after the remote NIC acks the message, but before the data may have been placed in memory.  Delivery complete likely means that the data is in memory.