<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>Hello All,</p>
<p><br>
</p>
<p>I get the following error when I try to run a benchmark test on a node with 256GB of RAM.</p>
<p><br>
</p>
<p><br>
</p>
<div>WARNING: It appears that your OpenFabrics subsystem is configured to only<br>
allow registering part of your physical memory. This can cause MPI jobs to<br>
run with erratic performance, hang, and/or crash.<br>
<br>
This may be caused by your OpenFabrics vendor limiting the amount of<br>
physical memory that can be registered. You should investigate the<br>
relevant Linux kernel module parameters that control how much physical<br>
memory can be registered, and increase them to allow registering all<br>
physical memory on your machine.<br>
<br>
See this Open MPI FAQ item for more information on these Linux kernel module<br>
parameters:<br>
<br>
http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages<br>
<br>
Local host: compute-0-3.local<br>
Registerable memory: 32768 MiB<br>
Total memory: 262109 MiB<br>
<br>
Your MPI job will continue, but may be behave poorly and/or hang.<br>
<br>
This only happens when I try it on nodes with more than 32GB of memory. Some searching around has lead me to the following bug report which seems to be the issue in my case too, because log_num_mtt on the node is 0 and log_mtts_per_seg is 3.<br>
<br>
<a id="LPlnk54426" href="https://bugzilla.redhat.com/show_bug.cgi?id=1171626" class="OWAAutoLink">https://bugzilla.redhat.com/show_bug.cgi?id=1171626</a><br>
<br>
The bug report suggests loading mlx5_core as a possible workaround.<br>
<br>
Any suggestions or other workarounds to make this work with mlx4_core?<br>
<br>
Appreciate any help I can get!<br>
<br>
Regards<br>
Krishna<br>
</div>
<p></p>
</div>
</body>
</html>