<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#467886;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">The sentence beginning “Since uncontended locks FI_THREAD_SAFE…” should read “Since uncontended locks aren’t very expensive, FI_THREAD_SAFE…”<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:"Calibri",sans-serif;mso-ligatures:none">From:</span></b><span style="font-family:"Calibri",sans-serif;mso-ligatures:none"> ofiwg <ofiwg-bounces@lists.openfabrics.org>
<b>On Behalf Of </b>Byrne, John (Labs)<br>
<b>Sent:</b> Wednesday, June 5, 2024 4:24 PM<br>
<b>To:</b> Xiong, Jianxin <jianxin.xiong@intel.com>; ofiwg@lists.openfabrics.org<br>
<b>Subject:</b> Re: [ofiwg] Clarification of definition of FI_THREAD_DOMAIN<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks. I couldn’t find a statement to that effect and every time I read about FI_THREAD_DOMAIN, I just kept assuming a single domain with multiple endpoints and thinking what a horrible idea that would be. Since uncontended locks FI_THREAD_SAFE
is certainly the simplest way to go assuming the only frequently used locks are on the endpoint and completion paths. If you have a MR Cache and are actively using it, then its locking gets annoying as things scale up, though.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">John<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:"Calibri",sans-serif;mso-ligatures:none">From:</span></b><span style="font-family:"Calibri",sans-serif;mso-ligatures:none"> Xiong, Jianxin <<a href="mailto:jianxin.xiong@intel.com">jianxin.xiong@intel.com</a>>
<br>
<b>Sent:</b> Wednesday, June 5, 2024 3:35 PM<br>
<b>To:</b> Byrne, John (Labs) <<a href="mailto:john.l.byrne@hpe.com">john.l.byrne@hpe.com</a>>;
<a href="mailto:ofiwg@lists.openfabrics.org">ofiwg@lists.openfabrics.org</a><br>
<b>Subject:</b> RE: Clarification of definition of FI_THREAD_DOMAIN<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Your understanding is correct. The recommendation is based on how feasible to have a lockless implementation in the providers. That also matches with how middleware like MPI is doing today.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Does FI_THREAD_COMPLETION fits better with the multi-threaded RMA use case? It is recommended for scalable endpoints because that’s when this threading model is more likely supported by the provider. But using that with regular endpoint
is totally fine if available.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">For simplicity at the user end, maybe just go with FI_THREAD_SAFE.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-Jianxin <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:"Calibri",sans-serif;mso-ligatures:none">From:</span></b><span style="font-family:"Calibri",sans-serif;mso-ligatures:none"> ofiwg <<a href="mailto:ofiwg-bounces@lists.openfabrics.org">ofiwg-bounces@lists.openfabrics.org</a>>
<b>On Behalf Of </b>Byrne, John (Labs)<br>
<b>Sent:</b> Wednesday, June 5, 2024 10:10 AM<br>
<b>To:</b> <a href="mailto:ofiwg@lists.openfabrics.org">ofiwg@lists.openfabrics.org</a><br>
<b>Subject:</b> [ofiwg] Clarification of definition of FI_THREAD_DOMAIN<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><b><i><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#333333;mso-ligatures:none">FI_THREAD_DOMAIN</span></i></b><b><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#333333;mso-ligatures:none"><o:p></o:p></span></b></p>
<p class="MsoNormal" style="margin-left:.5in;line-height:15.0pt;background:white">
<span style="font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#333333;mso-ligatures:none">A domain serialization model requires applications to serialize access to all objects belonging to a domain.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My immediate take on this definition is that if I am multi-threading I have to have a single lock that I use to access any object belonging to a fi_domain instance; which seems like a terrible idea for multi-threading. However, in Jianxin’s
2.0 API update at the workshop <a href="https://www.openfabrics.org/wp-content/uploads/2024-workshop/2024-workshop-presentations/session-1.pdf">https://www.openfabrics.org/wp-content/uploads/2024-workshop/2024-workshop-presentations/session-1.pdf</a>, it says: “Recommend FI_THREAD_DOMAIN for multi-thread app with regular endpoint.” If my interpretation of the meaning of FI_THREAD_DOMAIN is correct,
then the only way this makes sense to me is for the expectation to be that a unique fi_domain instance and endpoint be created for each thread. Is this correct or is there something I’m misunderstanding? If it is correct, then there are some painful implications
for multi-threading RMA.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">John Byrne<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>