<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>Howard</p>
<p><br>
</p>
<p>From the phrasing "<span>You are hitting a limitation with the ancient kdreg device driver. It may be best to not use it for your libfabric app.</span>" is there anything I can do about it. I can see that there is a udreg directory in /opt/cray - is there
anything I can replace the kdreg stuff with?</p>
<p><br>
</p>
<p>Thanks</p>
<p><br>
</p>
<p>JB</p>
<p><br>
</p>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Libfabric-users <libfabric-users-bounces@lists.openfabrics.org> on behalf of Biddiscombe, John A. <biddisco@cscs.ch><br>
<b>Sent:</b> 17 June 2020 17:26:29<br>
<b>To:</b> Howard Pritchard<br>
<b>Cc:</b> libfabric-users@lists.openfabrics.org<br>
<b>Subject:</b> Re: [libfabric-users] Error allocating domain</font>
<div> </div>
</div>
<div><style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p><span>my config line has always been this (apart from the debug). It has worked for several years until a recent system maintenance.change or something of that kind. (Nobody here claims to have changed anything significant)<br>
</span></p>
<p><span><br>
</span></p>
<p><span>./configure --disable-verbs --disable-sockets --disable-usnic --disable-udp --disable-rxm --disable-rxd --disable-shm --disable-mrail --disable-tcp --disable-perf --disable-rstream --enable-gni --prefix=/apps/daint/UES/biddisco/gcc/8.3.0/libfabric
CC=/opt/cray/pe/craype/default/bin/cc CFLAGS=-fPIC LDFLAGS=-ldl --no-recursion --enable-debug</span></p>
<p><span><br>
</span></p>
<p><span>JB</span><br>
</p>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Howard Pritchard <hppritcha@gmail.com><br>
<b>Sent:</b> 17 June 2020 17:20:21<br>
<b>To:</b> Biddiscombe, John A.<br>
<b>Cc:</b> libfabric-users@lists.openfabrics.org<br>
<b>Subject:</b> Re: [libfabric-users] Error allocating domain</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi John,
<div><br>
</div>
<div>You are hitting a limitation with the ancient kdreg device driver. It may be best to not use it for your libfabric app. What are the configure options you're using for building libfabric?</div>
<div><br>
</div>
<div>Howard</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Am Di., 16. Juni 2020 um 10:34 Uhr schrieb Biddiscombe, John A. <<a href="mailto:biddisco@cscs.ch">biddisco@cscs.ch</a>>:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div id="gmail-m_-1715228889037911668divtagdefaultwrapper" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<p></p>
<div>I've got this log when I dump out my own messages, and also enable debugging in libfabric - can anyone tell what's wrong from the message. Code that used to work seems to have stopped. I upgraded to libfabric 1.10.1 tag and rebuilt, but it didn't change.</div>
<div><br>
</div>
<div>The only thing that springs to mind is that the application is also using MPI on the cray at the same time, so when this code is called, mpi_init would have already been called, and perhaps somehow the nic is inaccessible - hence the error. I'm sure it
used to work - and if I use ranks = 1 - it runs - so perhaps mpi detects just one rank and does no initialization, but when I use N>1 ranks, it dies. Any suggestions welcome. Thanks</div>
<div><br>
</div>
<div>JB</div>
<div><br>
</div>
<div><br>
</div>
<div><DEB> 0000056511 0x2aaaaab2dec0 cpu 000 nid00219(0) CONTROL Allocating domain <br>
libfabric:69061:gni:core:_gnix_ref_init():254<debug> [69061:1] 0x8579d8 refs 1<br>
libfabric:69061:core:core:fi_fabric_():1154<info> Opened fabric: gni<br>
libfabric:69061:gni:domain:gnix_domain_open():579<trace> [69061:1] <br>
libfabric:69061:gni:fabric:gnix_domain_open():591<info> [69061:1] failed to find authorization key, creating new authorization key<br>
libfabric:69061:gni:domain:_gnix_auth_key_enable():347<info> [69061:1] pkey=dd920000 ptag=14 key_partition_size=0 key_offset=0 enabled<br>
libfabric:69061:gni:domain:gnix_domain_open():597<info> [69061:1] authorization key=0x857a10 ptag 14 cookie 0xdd920000<br>
libfabric:69061:gni:mr:_gnix_notifier_open():88<warn> [69061:1] kdreg device open failed: Device or resource busy<br>
<ERR> 0000056576 0x2aaaaab2dec0 cpu 000 nid00219(0) ERROR__ fi_domain : Device or resource busy</div>
<br>
<p></p>
</div>
</div>
_______________________________________________<br>
Libfabric-users mailing list<br>
<a href="mailto:Libfabric-users@lists.openfabrics.org" target="_blank">Libfabric-users@lists.openfabrics.org</a><br>
<a href="https://lists.openfabrics.org/mailman/listinfo/libfabric-users" rel="noreferrer" target="_blank">https://lists.openfabrics.org/mailman/listinfo/libfabric-users</a><br>
</blockquote>
</div>
</div>
</div>
</body>
</html>