<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;}
@font-face
{font-family:"\@DengXian";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#467886;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Your understanding is correct. The recommendation is based on how feasible to have a lockless implementation in the providers. That also matches with how middleware like MPI is doing today.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Does FI_THREAD_COMPLETION fits better with the multi-threaded RMA use case? It is recommended for scalable endpoints because that’s when this threading model is more likely supported by the provider. But using that with regular endpoint
is totally fine if available.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">For simplicity at the user end, maybe just go with FI_THREAD_SAFE.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-Jianxin <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:"Calibri",sans-serif;mso-ligatures:none">From:</span></b><span style="font-family:"Calibri",sans-serif;mso-ligatures:none"> ofiwg <ofiwg-bounces@lists.openfabrics.org>
<b>On Behalf Of </b>Byrne, John (Labs)<br>
<b>Sent:</b> Wednesday, June 5, 2024 10:10 AM<br>
<b>To:</b> ofiwg@lists.openfabrics.org<br>
<b>Subject:</b> [ofiwg] Clarification of definition of FI_THREAD_DOMAIN<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><b><i><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#333333;mso-ligatures:none">FI_THREAD_DOMAIN</span></i></b><b><span style="font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#333333;mso-ligatures:none"><o:p></o:p></span></b></p>
<p class="MsoNormal" style="margin-left:.5in;line-height:15.0pt;background:white">
<span style="font-size:10.5pt;font-family:"Helvetica",sans-serif;color:#333333;mso-ligatures:none">A domain serialization model requires applications to serialize access to all objects belonging to a domain.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My immediate take on this definition is that if I am multi-threading I have to have a single lock that I use to access any object belonging to a fi_domain instance; which seems like a terrible idea for multi-threading. However, in Jianxin’s
2.0 API update at the workshop <a href="https://www.openfabrics.org/wp-content/uploads/2024-workshop/2024-workshop-presentations/session-1.pdf">
https://www.openfabrics.org/wp-content/uploads/2024-workshop/2024-workshop-presentations/session-1.pdf</a>, it says: “Recommend FI_THREAD_DOMAIN for multi-thread app with regular endpoint.” If my interpretation of the meaning of FI_THREAD_DOMAIN is correct,
then the only way this makes sense to me is for the expectation to be that a unique fi_domain instance and endpoint be created for each thread. Is this correct or is there something I’m misunderstanding? If it is correct, then there are some painful implications
for multi-threading RMA.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">John Byrne<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>