[libfabric-users] question about message ordering

[Hartman, John]

Thank you, this helps a lot. I overlooked fi_ep_attr::max_order_waw_size, I’ll have to pay attention to that.

John

From: Hefty, Sean <sean.hefty at intel.com>
Date: Tuesday, February 8, 2022 at 10:49 AM
To: Hartman, John <john.hartman at hpe.com>, libfabric-users at lists.openfabrics.org <Libfabric-users at lists.openfabrics.org>
Subject: RE: question about message ordering
> I wanted to confirm my understanding of message ordering. The man page for FI_ORDER_WAW
> (for example) says
>
> FI_ORDER_WAW
>
> Write after write. If set, RMA and atomic write operations are transmitted in the order
> submitted relative to other RMA and atomic write operations. If not set, RMA and atomic
> writes may be transmitted out of order from their submission.
>
> Does the term “submitted” apply to all methods of initiating an RMA, e.g. fi_inject? In
> other words, if the provider supports FI_ORDER_WAW can I use fi_inject to send back-to-
> back RMA writes without worrying that they will be received out-of-order?

Mostly true.  The initiator will transfer the RMA writes in order to the target, regardless of which API was used.  Assuming that the target also has the same ordering, the RMA writes will be processed in order.

But also see fi_ep_attr::max_order_waw_size.  This field indicates whether the data from the first write will be written at the target before the data from the second write.  If max_order_waw_size == max_msg_size, the data is completely ordered.  If max_order_waw_size < max_msg_size, only data from smaller RMA writes will be completely ordered.

The distinction between message ordering (FI_ORDER_WAW) and data ordering is subtle.  I find it helps if you consider any completion event generated at the target.  WAW ordering means that the 1st write will generate a completion at the target before the 2nd write.  Data ordering says that the target cannot update memory targeted by the 2nd write until the 1st write is complete.

Together, these control the implementation and how it handles lost packets, out of order packets, and whether multiple paths through the network can be efficiently used.

- Sean
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20220208/bff60a23/attachment-0001.htm>