<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>Howard</p>
<p><br>
</p>
<p>Thanks for that link and your reply. I did waste a bit of time experimenting earlier, but soon found that leaving the mr_mode empty resulted in an error as well. I will stick to the MR_BASIC version and leave my memory management code in place. For a while
I was thinking I could remove it all, but I guess it'll have to wait for a few more releases - or a new machine.</p>
<p><br>
</p>
<p>Cheers</p>
<p><br>
</p>
<p>JB</p>
<p><br>
</p>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Howard Pritchard <hppritcha@gmail.com><br>
<b>Sent:</b> 31 August 2020 17:57:37<br>
<b>To:</b> Biddiscombe, John A.<br>
<b>Cc:</b> libfabric-users@lists.openfabrics.org<br>
<b>Subject:</b> Re: [libfabric-users] FI_MR_SCALABLE on GNI</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi John,
<div><br>
</div>
<div>Interesting you ask about this. The answer is that the matrix is only partially correct. The Aries hardware doesn't really support Mr. Scalable - it supports a very constrained sort of Mr. Scalable using FI_MR_MMU_NOTIFY.</div>
<div>There are other problems with the way it was implemented as well. For GNI provider, you'll do better sticking with FI_MR_BASIC. See the patches we're applying to Open MPI for this issue.</div>
<div><br>
</div>
<div><br>
</div>
<div>See <a href="https://github.com/ofiwg/libfabric/issues/6194">https://github.com/ofiwg/libfabric/issues/6194</a></div>
<div><br>
</div>
<div>Howard</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Am Mo., 31. Aug. 2020 um 05:00 Uhr schrieb Biddiscombe, John A. <<a href="mailto:biddisco@cscs.ch">biddisco@cscs.ch</a>>:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div id="gmail-m_1064429451713881024divtagdefaultwrapper" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<p></p>
<div>Dear List</div>
<div><br>
</div>
<div>libfabric:16321:gni:core:ofi_check_mr_mode():545<info> Invalid memory registration mode<br>
libfabric:16321:gni:core:ofi_check_mr_mode():546<info> Expected: FI_MR_BASIC, FI_MR_MMU_NOTIFY<br>
libfabric:16321:gni:core:ofi_check_mr_mode():546<info> Given: FI_MR_SCALABLE<br>
libfabric:16321:gni:domain:_gnix_ep_getinfo():597<info> [16321:1] failed ofi_check_mr_mode, ret=-61</div>
<p></p>
<p><br>
</p>
<p>I'm using the libfabric 1.11 tag from the git repo and according to the feature matrix, scalable memory registration is supported on gni, however, I get the above error in the logs when I ask for it in using
<br>
</p>
<p> <span>fabric_hints_->domain_attr->mr_mode = FI_MR_SCALABLE;</span></p>
<p><br>
</p>
<p>Is the matrix incorrect, or should I keep playing with flags in case I've messed something up elsewhere?</p>
<p><br>
</p>
<p>Thanks</p>
<p><br>
</p>
<p>JB<br>
</p>
</div>
</div>
_______________________________________________<br>
Libfabric-users mailing list<br>
<a href="mailto:Libfabric-users@lists.openfabrics.org" target="_blank">Libfabric-users@lists.openfabrics.org</a><br>
<a href="https://lists.openfabrics.org/mailman/listinfo/libfabric-users" rel="noreferrer" target="_blank">https://lists.openfabrics.org/mailman/listinfo/libfabric-users</a><br>
</blockquote>
</div>
</div>
</body>
</html>