<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:12.0pt">Thank you, this helps a lot. I overlooked
</span>fi_ep_attr::max_order_waw_size, I’ll have to pay attention to that.<span style="font-size:12.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">John</span><span style="font-size:12.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Hefty, Sean <sean.hefty@intel.com><br>
<b>Date: </b>Tuesday, February 8, 2022 at 10:49 AM<br>
<b>To: </b>Hartman, John <john.hartman@hpe.com>, libfabric-users@lists.openfabrics.org <Libfabric-users@lists.openfabrics.org><br>
<b>Subject: </b>RE: question about message ordering<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">> I wanted to confirm my understanding of message ordering. The man page for FI_ORDER_WAW<br>
> (for example) says<br>
> <br>
> FI_ORDER_WAW<br>
> <br>
> Write after write. If set, RMA and atomic write operations are transmitted in the order<br>
> submitted relative to other RMA and atomic write operations. If not set, RMA and atomic<br>
> writes may be transmitted out of order from their submission.<br>
> <br>
> Does the term “submitted” apply to all methods of initiating an RMA, e.g. fi_inject? In<br>
> other words, if the provider supports FI_ORDER_WAW can I use fi_inject to send back-to-<br>
> back RMA writes without worrying that they will be received out-of-order?<br>
<br>
Mostly true. The initiator will transfer the RMA writes in order to the target, regardless of which API was used. Assuming that the target also has the same ordering, the RMA writes will be processed in order.<br>
<br>
But also see fi_ep_attr::max_order_waw_size. This field indicates whether the data from the first write will be written at the target before the data from the second write. If max_order_waw_size == max_msg_size, the data is completely ordered. If max_order_waw_size
< max_msg_size, only data from smaller RMA writes will be completely ordered. <br>
<br>
The distinction between message ordering (FI_ORDER_WAW) and data ordering is subtle. I find it helps if you consider any completion event generated at the target. WAW ordering means that the 1st write will generate a completion at the target before the 2nd
write. Data ordering says that the target cannot update memory targeted by the 2nd write until the 1st write is complete.<br>
<br>
Together, these control the implementation and how it handles lost packets, out of order packets, and whether multiple paths through the network can be efficiently used.<br>
<br>
- Sean<o:p></o:p></p>
</div>
</div>
</body>
</html>