<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Sean --</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I sent a follow-up this morning, which you may not have seen yet. In that I hypothesized that this was due to the CQ entry ordering not being the same as the message ordering in the buffer because I'd thrown FI_ORDER_SAS on both the sending and receiving endpoints,
and the FI_MULTI_RECEIVE CQ entry was effectively in buffer order rather than CQ order.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
To answer your other questions, this was with the 1.10.1 release tarball and I'm only using one multi-receive buffer at a time.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
greg</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Hefty, Sean <sean.hefty@intel.com><br>
<b>Sent:</b> Wednesday, June 24, 2020 1:04 PM<br>
<b>To:</b> Titus, Greg <gregory.titus@hpe.com>; libfabric-users@lists.openfabrics.org <libfabric-users@lists.openfabrics.org><br>
<b>Subject:</b> RE: multi-receive CQ entry with FI_MULTI_RECV, not the last one read by fi_cq_read()?</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">> > I post a multi-receive buffer via fi_recvmsg(..., FI_MULTI_RECV) and then process it<br>
> by<br>
> > reading CQ entries which refer to the messages landing there. When I see a CQ entry<br>
> > with FI_MULTI_RECV in its flags I re-post that same multi-receive buffer with<br>
> > fi_recvmsg() again. Based on reading the man pages, I've put an assertion in my CQ<br>
> > entry processing that if an entry has FI_MULTI_RECV in its flags, that entry must be<br>
> > the last one my fi_cq_read() read. Essentially, this assertion is to confirm my<br>
> > understanding that nothing can be placed in the multi-receive buffer after the<br>
> provider<br>
> > releases it.<br>
> <br>
> This should be the case. What version of the library are you using?<br>
> <br>
> Are you posting more than 1 multi-recv buffer? There were issues in older code where<br>
> providers were reposting multi-recv buffers to the end of the receive queue. So, if<br>
> there were 2 buffers posted, received data would intermingle between them. (Data<br>
> ordering was unaffected).<br>
<br>
In addition, there was also an issue in the multi-recv posting path not handling unexpected messages correctly. I don't remember which version of libfabric correct the above issues, but they should be fixed in the upstream code.<br>
<br>
- Sean<br>
</div>
</span></font></div>
</body>
</html>