<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Hi John,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
I know your question is aimed at Howard, but I can offer another data point and an example of a software stack working around this. I've never gotten kdreg to work in executables that are also using Cray's MPI; they conflict. If you want to use udreg as an
alternative, then you'll need to do two things:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
a) disable kdreg support in libfabric at build time (as in this spack package here:
<a href="https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/libfabric/package.py#L94">
https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/libfabric/package.py#L94</a>)</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
b) explicitly enable and configure udreg outside of libfabric (as in the Mercury libfabric plugin here:
<a href="https://github.com/mercury-hpc/mercury/blob/master/src/na/na_ofi.c#L1778">
https://github.com/mercury-hpc/mercury/blob/master/src/na/na_ofi.c#L1778</a>) <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
This configuration is stable for us and works fine whether Cray MPI is present or not. I'll defer to Howard about the technical implications, though
<span id="🙂">🙂</span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span id="🙂">thanks,</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span id="🙂">-Phil</span><br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Libfabric-users <libfabric-users-bounces@lists.openfabrics.org> on behalf of Biddiscombe, John A. <biddisco@cscs.ch><br>
<b>Sent:</b> Wednesday, June 17, 2020 1:32 PM<br>
<b>To:</b> Howard Pritchard <hppritcha@gmail.com><br>
<b>Cc:</b> libfabric-users@lists.openfabrics.org <libfabric-users@lists.openfabrics.org><br>
<b>Subject:</b> Re: [libfabric-users] Error allocating domain</font>
<div> </div>
</div>
<div><style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<p>Howard</p>
<p><br>
</p>
<p>From the phrasing "<span>You are hitting a limitation with the ancient kdreg device driver. It may be best to not use it for your libfabric app.</span>" is there anything I can do about it. I can see that there is a udreg directory in /opt/cray - is there
anything I can replace the kdreg stuff with?</p>
<p><br>
</p>
<p>Thanks</p>
<p><br>
</p>
<p>JB</p>
<p><br>
</p>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Libfabric-users <libfabric-users-bounces@lists.openfabrics.org> on behalf of Biddiscombe, John A. <biddisco@cscs.ch><br>
<b>Sent:</b> 17 June 2020 17:26:29<br>
<b>To:</b> Howard Pritchard<br>
<b>Cc:</b> libfabric-users@lists.openfabrics.org<br>
<b>Subject:</b> Re: [libfabric-users] Error allocating domain</font>
<div> </div>
</div>
<div><style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<p><span>my config line has always been this (apart from the debug). It has worked for several years until a recent system maintenance.change or something of that kind. (Nobody here claims to have changed anything significant)<br>
</span></p>
<p><span><br>
</span></p>
<p><span>./configure --disable-verbs --disable-sockets --disable-usnic --disable-udp --disable-rxm --disable-rxd --disable-shm --disable-mrail --disable-tcp --disable-perf --disable-rstream --enable-gni --prefix=/apps/daint/UES/biddisco/gcc/8.3.0/libfabric
CC=/opt/cray/pe/craype/default/bin/cc CFLAGS=-fPIC LDFLAGS=-ldl --no-recursion --enable-debug</span></p>
<p><span><br>
</span></p>
<p><span>JB</span><br>
</p>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Howard Pritchard <hppritcha@gmail.com><br>
<b>Sent:</b> 17 June 2020 17:20:21<br>
<b>To:</b> Biddiscombe, John A.<br>
<b>Cc:</b> libfabric-users@lists.openfabrics.org<br>
<b>Subject:</b> Re: [libfabric-users] Error allocating domain</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi John,
<div><br>
</div>
<div>You are hitting a limitation with the ancient kdreg device driver. It may be best to not use it for your libfabric app. What are the configure options you're using for building libfabric?</div>
<div><br>
</div>
<div>Howard</div>
<div><br>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">Am Di., 16. Juni 2020 um 10:34 Uhr schrieb Biddiscombe, John A. <<a href="mailto:biddisco@cscs.ch">biddisco@cscs.ch</a>>:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div id="x_gmail-m_-1715228889037911668divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif">
<p></p>
<div>I've got this log when I dump out my own messages, and also enable debugging in libfabric - can anyone tell what's wrong from the message. Code that used to work seems to have stopped. I upgraded to libfabric 1.10.1 tag and rebuilt, but it didn't change.</div>
<div><br>
</div>
<div>The only thing that springs to mind is that the application is also using MPI on the cray at the same time, so when this code is called, mpi_init would have already been called, and perhaps somehow the nic is inaccessible - hence the error. I'm sure it
used to work - and if I use ranks = 1 - it runs - so perhaps mpi detects just one rank and does no initialization, but when I use N>1 ranks, it dies. Any suggestions welcome. Thanks</div>
<div><br>
</div>
<div>JB</div>
<div><br>
</div>
<div><br>
</div>
<div><DEB> 0000056511 0x2aaaaab2dec0 cpu 000 nid00219(0) CONTROL Allocating domain <br>
libfabric:69061:gni:core:_gnix_ref_init():254<debug> [69061:1] 0x8579d8 refs 1<br>
libfabric:69061:core:core:fi_fabric_():1154<info> Opened fabric: gni<br>
libfabric:69061:gni:domain:gnix_domain_open():579<trace> [69061:1] <br>
libfabric:69061:gni:fabric:gnix_domain_open():591<info> [69061:1] failed to find authorization key, creating new authorization key<br>
libfabric:69061:gni:domain:_gnix_auth_key_enable():347<info> [69061:1] pkey=dd920000 ptag=14 key_partition_size=0 key_offset=0 enabled<br>
libfabric:69061:gni:domain:gnix_domain_open():597<info> [69061:1] authorization key=0x857a10 ptag 14 cookie 0xdd920000<br>
libfabric:69061:gni:mr:_gnix_notifier_open():88<warn> [69061:1] kdreg device open failed: Device or resource busy<br>
<ERR> 0000056576 0x2aaaaab2dec0 cpu 000 nid00219(0) ERROR__ fi_domain : Device or resource busy</div>
<br>
<p></p>
</div>
</div>
_______________________________________________<br>
Libfabric-users mailing list<br>
<a href="mailto:Libfabric-users@lists.openfabrics.org" target="_blank">Libfabric-users@lists.openfabrics.org</a><br>
<a href="https://lists.openfabrics.org/mailman/listinfo/libfabric-users" rel="noreferrer" target="_blank">https://lists.openfabrics.org/mailman/listinfo/libfabric-users</a><br>
</blockquote>
</div>
</div>
</div>
</div>
</body>
</html>