[libfabric-users] question about "default" FI_DELIVERY_COMPLETE semantics for RMA reads and fetching atomics

[Titus, Greg]

Following up my own question, I just realized that the fi_atomic(3) man page is explicit about this: fetching atomic ops always use FI_DELIVER_COMPLETE for their results, no matter what.  So my question really only applies to RMA reads.

greg

________________________________
From: Libfabric-users <libfabric-users-bounces at lists.openfabrics.org> on behalf of Titus, Greg <gregory.titus at hpe.com>
Sent: Monday, February 10, 2020 1:08 PM
To: libfabric-users at lists.openfabrics.org <libfabric-users at lists.openfabrics.org>
Subject: [libfabric-users] question about "default" FI_DELIVERY_COMPLETE semantics for RMA reads and fetching atomics

Greetings --

The fi_cq(3) man page says that
[FI_DELIVERY_COMPLETE] applies only to reliable endpoints. For operations that return data to the initiator, such as RMA read or atomic-fetch, the source endpoint is also considered a destination endpoint. This is the default completion mode for such operations.

Does "default completion mode" in that last sentence really mean "default", in that it's the completion mode that will be used if and only if no FI_{INJECT,TRANSMIT,DELIVERY,...}_COMPLETE flag is present in the hints to fi_getinfo() and/or the flags to fi_readmsg() etc.?  Or does it really mean that delivery_complete is guaranteed for transactions that produce local results, such that even if for example FI_TRANSMIT_COMPLETE is given in the hints, RMA reads and fetching atomics will nevertheless effectively use delivery_complete?

(I'm pretty sure it's the former, but thought I should check.)

If it is indeed the former, when an RMA read or fetching atomic is done and FI_TRANSMIT_COMPLETE is in force, is there any way to tell when the data has arrived?

thanks,
greg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20200210/7d67c0c2/attachment.htm>