<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<div id="divtagdefaultwrapper" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, "EmojiFont", "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
<p>Sean et al</p>
<p><br>
</p>
<p>I'm just resurrecting an old thread for clarification</p>
<p><br>
</p>
><font size="2"><span style="font-size:10pt;">If the transfer is large, it won't complete immediately and the buffering will occur on the send side. Only the tagged information will be transferred. Once the receiver posts a receive with the correct tag, the
data will be retrieved.</span></font></div>
<div style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, "EmojiFont", "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
<br>
</div>
<div style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, "EmojiFont", "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
<p>Does this mean that if I send a large tagged message (What is the threshold?, then only the tag info is sent initially and
<br>
</p>
<p>a) if a posted receive buffer with a tag match is already present, the receiving side will do an RMA get from the send side buffer to the receive buffer (and then both sides get a completion)</p>
<p>b) if no receive buffer is posted yet, the send side will not post a completion, but when the receive side does post a matching buffer, an RMA get will finish the job and both sides complete.</p>
<p><br>
</p>
<p>The reason I ask is because I handle larger messages (>threshold) using RMA - but if libfabric is doing this internally, I can drop this completely for tagged messages and simply use fi_trecv and fi_tsend regardless of size.</p>
<p><br>
</p>
<p>what I really mean is - Is there any advantage to me sending the memory registration info in a small (unexpected) message and then doing an RMA get from the remote side myself when the tagged receive is setup?
<br>
</p>
</div>
<div style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, "EmojiFont", "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
<br>
</div>
<div style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, "EmojiFont", "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
Thanks</div>
<div style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, "EmojiFont", "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
<br>
</div>
<div style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, "EmojiFont", "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
JB<br>
<div style="color: rgb(0, 0, 0);">
<div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Hefty, Sean <sean.hefty@intel.com><br>
<b>Sent:</b> 16 March 2020 17:53<br>
<b>To:</b> Chris Dolan; Biddiscombe, John A.<br>
<b>Cc:</b> libfabric-users@lists.openfabrics.org<br>
<b>Subject:</b> RE: [libfabric-users] Tagged message question</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">> Who provided the memory that the received message went into between being<br>
> received and my posting the recv buffer? Did libfabric supply some memory buffer 'on<br>
> the fly' that the tagged message went into?<br>
<br>
Adding to Chris' comments, these are known as 'unexpected messages'. They are a common MPI problem and nearly impossible to avoid, even for well-designed apps.<br>
<br>
Most commonly available hardware lack the ability to perform tag matching. As a result, rxm buffers all receives before deciding how to process it. If you're using RDM endpoints, there's really no additional cost for using tagged transfers, versus untagged.<br>
<br>
If the transfer is large, it won't complete immediately and the buffering will occur on the send side. Only the tagged information will be transferred. Once the receiver posts a receive with the correct tag, the data will be retrieved.<br>
<br>
- Sean<br>
</div>
</span></font></div>
</div>
</div>
</body>
</html>