<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<meta content="text/html; charset=UTF-8">
<style type="text/css" style="">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<p>Verified - tags are 64bits on tcp and on gni 0xaaaaaaaaaaaaaaa</p>
<p><br>
</p>
<p>I just tested on GNI and I get exactly the same problem, my snippet of log looks like this</p>
<p></p>
<div><br>
</div>
<div><DEB> 0000279468 0x2aaaaab1d340 cpu --- nid00986(0) COMMUNI recv message buffer <- 00 tag 0x00761ff800000000<br>
<DEB> 0000279505 0x2aaaaab1d340 cpu --- nid00986(0) COMMUNI recv message buffer <- 00 tag 0x00761ff800000003<br>
<DEB> 0000279564 0x2aaaaab1d340 cpu --- nid00986(0) COMMUNI recv message buffer <- 00 tag 0x00761ff800000004<br>
<DEB> 0000279589 0x2aaaaab1d340 cpu --- nid00986(0) COMMUNI recv message buffer <- 00 tag 0x00761ff800000001<br>
<DEB> 0000279649 0x2aaaaab1d340 cpu --- nid00986(0) COMMUNI send message buffer -> 00 tag 0x00761ff800000000<br>
<DEB> 0000280477 0x2aaaaab1d340 cpu --- nid00986(0) COMMUNI send message buffer -> 00 tag 0x00761ff800000003<br>
<DEB> 0000280531 0x2aaaaab1d340 cpu --- nid00986(0) COMMUNI send message buffer -> 00 tag 0x00761ff800000004<br>
<DEB> 0000280583 0x2aaaaab1d340 cpu --- nid00986(0) COMMUNI send message buffer -> 00 tag 0x00761ff800000001<br>
<DEB> 0000280676 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL Completion txcq MSG tagged send completion 0x81bbe0<br>
<DEB> 0000280713 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL set_ready 0x00761ff800000000 send<br>
<DEB> 0000280795 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL Completion txcq MSG tagged send completion 0x81bcb0<br>
<DEB> 0000280803 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL set_ready 0x00761ff800000003 send<br>
<DEB> 0000280810 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL Completion txcq MSG tagged send completion 0x8242b0<br>
<DEB> 0000280817 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL set_ready 0x00761ff800000004 send<br>
<DEB> 0000280824 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL Completion txcq MSG tagged send completion 0x824370<br>
<DEB> 0000280830 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL set_ready 0x00761ff800000001 send<br>
<br>
</div>
when I use a separate endpoint for send/recv - <span>4 sends are matched with correct tags</span> - the recvs are not matched. (I only add the endpoint address of the receive endpoint to the AV, not the send endpoint)
<p></p>
<p>I am polling both tx/rx cq's</p>
<p><br>
</p>
<p>if I switch to a single endpoint for both - then everything works as expected using gni,sockets,tcp</p>
<div><DEB> 0000246689 0x2aaaaab1d340 cpu --- nid00986(0) COMMUNI recv message buffer <- 00 tag 0x00761f9800000000<br>
<DEB> 0000246721 0x2aaaaab1d340 cpu --- nid00986(0) COMMUNI recv message buffer <- 00 tag 0x00761f9800000003<br>
<DEB> 0000246742 0x2aaaaab1d340 cpu --- nid00986(0) COMMUNI recv message buffer <- 00 tag 0x00761f9800000004<br>
<DEB> 0000246762 0x2aaaaab1d340 cpu --- nid00986(0) COMMUNI recv message buffer <- 00 tag 0x00761f9800000001<br>
<DEB> 0000246806 0x2aaaaab1d340 cpu --- nid00986(0) COMMUNI send message buffer -> 00 tag 0x00761f9800000000<br>
<DEB> 0000247215 0x2aaaaab1d340 cpu --- nid00986(0) COMMUNI send message buffer -> 00 tag 0x00761f9800000003<br>
<DEB> 0000247251 0x2aaaaab1d340 cpu --- nid00986(0) COMMUNI send message buffer -> 00 tag 0x00761f9800000004<br>
<DEB> 0000247293 0x2aaaaab1d340 cpu --- nid00986(0) COMMUNI send message buffer -> 00 tag 0x00761f9800000001<br>
<DEB> 0000247382 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL Completion txcq MSG tagged send completion 0x7e7660<br>
<DEB> 0000247391 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL set_ready 0x00761f9800000000 send<br>
<DEB> 0000247399 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL Completion rxcq MSG tagged recv completion 0x7e6340<br>
<DEB> 0000247405 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL set_ready 0x00761f9800000000 recv<br>
<DEB> 0000247411 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL Completion txcq MSG tagged send completion 0x7e7730<br>
<DEB> 0000247418 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL set_ready 0x00761f9800000003 send<br>
<DEB> 0000247423 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL Completion rxcq MSG tagged recv completion 0x7e6230<br>
<DEB> 0000247430 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL set_ready 0x00761f9800000003 recv<br>
<DEB> 0000247435 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL Completion txcq MSG tagged send completion 0x7eb9e0<br>
<DEB> 0000247441 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL set_ready 0x00761f9800000004 send<br>
<DEB> 0000247446 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL Completion rxcq MSG tagged recv completion 0x7e6050<br>
<DEB> 0000247452 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL set_ready 0x00761f9800000004 recv<br>
<DEB> 0000247457 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL Completion txcq MSG tagged send completion 0x7ebaa0<br>
<DEB> 0000247464 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL set_ready 0x00761f9800000001 send<br>
<DEB> 0000247469 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL Completion rxcq MSG tagged recv completion 0x7bf770<br>
<DEB> 0000247475 0x2aaaaab1d340 cpu --- nid00986(0) CONTROL set_ready 0x00761f9800000001 recv<br>
<br>
</div>
<div>now we have 4 matched sends and 4 recvs - this test is using 1 rank only (sending to itselff)</div>
<div><br>
</div>
<div>I must have some mistake in my setup of the endpoints, but I just don't know what it might be - is there an example anywhere that uses
<span>different</span> tx/rx endpoints?</div>
<div><br>
</div>
<div>Many thanks for your patience</div>
<div><br>
</div>
<div>JB<br>
</div>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Hefty, Sean <sean.hefty@intel.com><br>
<b>Sent:</b> 12 February 2021 22:00:53<br>
<b>To:</b> Biddiscombe, John A.; libfabric-users@lists.openfabrics.org<br>
<b>Subject:</b> RE: Not receiving messages from other ranks</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">> I'm using 64bit tags and I get weird (=I can't explain it well) results if I put some<br>
> info like rank in the tag using bitmasks - then the sockets version stops working (but<br>
> I have no idea why making the tags more different/unique would be an issue (however,<br>
> mismatched tags would explain the data fail when tests are run with sockets - if the<br>
> wrong tag is being matched, we get errors, but if I change the tags, nothing matches -<br>
> should I only use 32bit tags?).<br>
<br>
Check the results from fi_getinfo. There's a tag mask somewhere in the attributes that indicates which tags are valid. I believe most providers are in the 60+ tag range, depending on other options, but I'm not sure about the full 64-bit range.<br>
<br>
- Sean<br>
</div>
</span></font>
</body>
</html>