<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hmm, interesting. My only ordering assertions are that I have <span>FI_ORDER_RMA_RAW</span>|FI_ORDER_SAS<span>|FI_ORDER_SAW in both the tx_attr and rx_attr msg_order members of the hints for my fi_getinfo(). In the tx_attr and rx_attr op_flags I've just got
FI_COMPLETION. But the CQ entry with the FI_MULTI_RECV flag set was entry 1 out of a range of 0..4, as I said. Definitely not the last one.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>greg</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span><br>
</span></div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Hefty, Sean <sean.hefty@intel.com><br>
<b>Sent:</b> Wednesday, June 24, 2020 1:39 PM<br>
<b>To:</b> Titus, Greg <gregory.titus@hpe.com>; libfabric-users@lists.openfabrics.org <libfabric-users@lists.openfabrics.org><br>
<b>Subject:</b> RE: multi-receive CQ entry with FI_MULTI_RECV, not the last one read by fi_cq_read()?</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">> I sent a follow-up this morning, which you may not have seen yet. In that I<br>
> hypothesized that this was due to the CQ entry ordering not being the same as the<br>
> message ordering in the buffer because I'd thrown FI_ORDER_SAS on both the sending and<br>
> receiving endpoints, and the FI_MULTI_RECEIVE CQ entry was effectively in buffer order<br>
> rather than CQ order.<br>
<br>
Completion order is separate from data order and not guaranteed unless FI_ORDER_STRICT has been enabled. I don't know how you structured your assert, but the completion with FI_MULTI_RECV flag set should be last, as it indicates that the buffer is safe to
repost.<br>
<br>
> To answer your other questions, this was with the 1.10.1 release tarball and I'm only<br>
> using one multi-receive buffer at a time.<br>
<br>
The fixes I was referring to are in that release.<br>
<br>
- Sean<br>
</div>
</span></font></div>
</body>
</html>