<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(32, 31, 30); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important">Thanks,
Sean. FI_MR_CACHE_MONITOR=memhooks failed in the same way. I'</span><span style="color: rgb(32, 31, 30); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px;">ll
file an Issue.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(32, 31, 30); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px;"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(32, 31, 30); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px;">greg</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(32, 31, 30); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px;"><br>
</span></div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Hefty, Sean <sean.hefty@intel.com><br>
<b>Sent:</b> Thursday, July 30, 2020 3:27 PM<br>
<b>To:</b> Titus, Greg <gregory.titus@hpe.com>; libfabric-users@lists.openfabrics.org <libfabric-users@lists.openfabrics.org><br>
<b>Subject:</b> RE: [libfabric-users] assertion failure, perhaps in registration cache, in 1.10.1 with verbs; ofi_rxm</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">I would open an github issue for this.<br>
<br>
You can also try using the other cache monitor (MEMHOOKS). That will also capture non-malloc based allocations (e.g. mmap), though you would need to use a custom build off master to pickup coverage for some allocations (sbrk I think).<br>
<br>
> I'm using the verbs;ofi_rxm provider with libfabric 1.10.1, on an IB-based Cray CS<br>
> system. I saw this in fi_mr(3):<br>
> <br>
> As a general rule, if hardware requires the FI_MR_LOCAL mode bit described above,<br>
> but this is not supported by the application, a memory registration cache may be in<br>
> use.<br>
> <br>
> <br>
> <br>
> I thought to myself, "Let's try it!" I set FI_MR_CACHE_MONITOR=userfaultfd, because my<br>
> application doesn't necessarily allocate all its memory through malloc() etc. I<br>
> removed FI_MR_LOCAL from my hints, while retaining (FI_MR_VIRT_ADDR | FI_MR_PROV_KEY |<br>
> FI_MR_ALLOCATED). My only other FI_* env var was FI_LOG_LEVEL=Warn. I verified that<br>
> I still got the verbs;ofi_rxm provider, and that FI_MR_LOCAL was clear in the returned<br>
> info. My 2-node test case ran properly, but then failed with the following assertion<br>
> on both nodes, in the call stack for fi_close(&ofi_domain->fid) (where ofi_domain is<br>
> the result of the fi_domain() call):<br>
> <br>
> a.out: .../prov/util/src/util_buf.c:220: ofi_bufpool_destroy: Assertion `(pool-<br>
> >attr.flags & OFI_BUFPOOL_NO_TRACK) || (buf_region->use_cnt == 0)' failed.<br>
> <br>
> <br>
> <br>
> <br>
> In the resulting core file, I find that it's the second clause (buf_region->use_cnt ==<br>
> 0) of the assertion that's false. That use_cnt is 1 (one). No output seemed to result<br>
> from my having set FI_LOG_LEVEL=Warn.<br>
> <br>
> <br>
> What's going on? Do I need to do some other setup to use the registration cache? Have<br>
> I failed to fi_close() something? (I looked and nothing jumped out at me, plus this<br>
> exact binary runs fine if I include FI_MR_LOCAL in the hints and don't change anything<br>
> else.)<br>
> <br>
> <br>
> thanks,<br>
> greg<br>
<br>
</div>
</span></font></div>
</body>
</html>