<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:595.0pt 842.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:14.0pt">OK. So now I’m sure. This morning I tested and attached the debugger and I see this horror<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><a href="https://gist.github.com/biddisco/95ea97941ad3a2bdeab1e875adfbda5a">https://gist.github.com/biddisco/95ea97941ad3a2bdeab1e875adfbda5a</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">(N threads are stuck on this lock and one that hold it has been suspended by HPX, so we have a deadlock scenario)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">So I’ll need to either block multiple threads from polling, or convert libfabric to use the hpx threading model.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt">JB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:14.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">Libfabric-users <libfabric-users-bounces@lists.openfabrics.org> on behalf of John Biddiscombe <biddisco@cscs.ch><br>
<b>Date: </b>Monday, 13 March 2017 at 00:16<br>
<b>To: </b>"libfabric-users@lists.openfabrics.org" <libfabric-users@lists.openfabrics.org><br>
<b>Subject: </b>[libfabric-users] Thread question about gni provider</span><span style="color:black;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Times New Roman""><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:14.0pt">Last question : I hope </span>
<span style="font-size:14.0pt;font-family:Wingdings">J</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:14.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:14.0pt">I have my parcelport running correctly using the gni provider and all the simple message sends and rdma reads are working as expected – however – with more complex examples, there are lockups.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:14.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:14.0pt">Is the GNI provider taking locks using pthread mutexes? I see them mentioned in the wait objects code, but I am not using any wait_sets directly and I wonder if they are present elsewhere.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:14.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:14.0pt">Unfortunately, the HPX runtime, uses lightweight threads - and OS level mutexes screw things up in a bad way, so I need to remove them. (The verbs API is entirely thread safe, so my code ran very nicely without
any problems – I had hoped the fabric port would do the same – I gave up on trying to use ucx for similar reasons, as it isn’t thread safe at the transport level).</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:14.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:14.0pt">Question : if GNI is using pthread mutexes and I replace the locks in libfabric GNI with HPX spinlocks and compile libfabric into the hpx project rather than building it outside – so that it becomes part of
the runtime – will the GNI layer work, or are there requirements that GNI api calls _<i>must</i>_ be made on certain threads? I seem to recall that GNI has some limitations in terms of threading and am worried that whatever I do, I’m going to have problems
with the locking or threading issues.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:14.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:14.0pt">Thanks once more for any guidance</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:14.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:14.0pt">JB</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:14.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:14.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Times New Roman";mso-fareast-language:EN-GB">--</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Times New Roman";mso-fareast-language:EN-GB">John Biddiscombe, email:biddisco @.at.@ cscs.ch</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Times New Roman";mso-fareast-language:EN-GB"><a href="http://www.cscs.ch/">http://www.cscs.ch/</a></span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Times New Roman";mso-fareast-language:EN-GB">CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="IT-CH" style="font-family:"Times New Roman";mso-fareast-language:EN-GB">Via Trevano 131, 6900 Lugano, Switzerland | Fax: +41 (91) 610.82.82</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</body>
</html>