<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
p.xmsonormal, li.xmsonormal, div.xmsonormal
{mso-style-name:x_msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.xapple-converted-space
{mso-style-name:x_apple-converted-space;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="xmsonormal" style="margin:0in;margin-bottom:.0001pt"><span style="font-size:12.0pt;color:black">I’d be interested in exposing an environment variable, flag, or general tunable in libfabric for applications to indicate whether they want to use huge
pages. Using the verbs provider on an internal development system, I’ve run into an issue where use of huge pages makes the memory registration function fail unless RDMAV_HUGEPAGES_SAFE is set. See</span><span class="xapple-converted-space"><span style="font-size:12.0pt;color:black"> </span></span><span style="font-size:12.0pt;color:black"><a href="https://lists.openfabrics.org/pipermail/ewg/2010-July/015609.html"><span style="color:#954F72">https://lists.openfabrics.org/pipermail/ewg/2010-July/015609.html</span></a></span><span class="xapple-converted-space"><span style="font-size:12.0pt;color:black"> </span></span><span style="font-size:12.0pt;color:black">for
context.<o:p></o:p></span></p>
<p class="xmsonormal" style="margin:0in;margin-bottom:.0001pt;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:12.0pt;color:black"> <o:p></o:p></span></p>
<p class="xmsonormal" style="margin:0in;margin-bottom:.0001pt;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:12.0pt;color:black">The general idea for the change would be a flag/environment variable which would control common core or provider memory allocation routines for buffers that would be registered with the NIC. If set, it would indicate
that the application intends to use huge pages and should take appropriate action. If not, the providers could assume that huge pages won’t be used. I had considered reusing the RDMAV_HUGEPAGES_SAFE environment variable, but I did not want to make conditional
behavior in common core code based on a variable specific to verbs. Especially if that variable could be generally set on a system and not specifically indicative of the application’s intended use of huge pages.<o:p></o:p></span></p>
<p class="xmsonormal" style="margin:0in;margin-bottom:.0001pt;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:12.0pt;color:black"> <o:p></o:p></span></p>
<p class="xmsonormal" style="margin:0in;margin-bottom:.0001pt;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:12.0pt;color:black">To me, it would be ideal if we can expose some knob to users to indicate expectations regarding the use of hugepages – regardless of what that knob is. The rationale for this is based on the non-zero cost of detecting
page sizes at registration time, especially for providers that are operating under the FI_LOCAL_MR mode bit. Common core code used by a provider has an effect on hugepage usage and a tunable for enabling or disabling hugepage usage could be useful for diagnostics,
debugging, and/or performance.<span class="apple-converted-space"> </span><o:p></o:p></span></p>
<p class="xmsonormal" style="margin:0in;margin-bottom:.0001pt;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:12.0pt;color:black"> <o:p></o:p></span></p>
<p class="xmsonormal" style="margin:0in;margin-bottom:.0001pt;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:12.0pt;color:black">I’m not a huge fan of environment variables, but I do see a lack of a solution for libfabric to indicate hugepage usage. Maybe it’s not necessary, but I wanted to start the discussion to see what other ideas are out
there.<o:p></o:p></span></p>
<p class="xmsonormal" style="margin:0in;margin-bottom:.0001pt;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:12.0pt;color:black"> <o:p></o:p></span></p>
<p class="xmsonormal" style="margin:0in;margin-bottom:.0001pt;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:12.0pt;color:black">For the verbs provider, I believe the simple first step is to set the environment variable to correct the failing behavior – and then determine how to handle huge pages support in general for libfabric in general,
if necessary. For verbs, use of the RDMAV_HUGEPAGES_SAFE variable tends to impact performance mostly after the rendezvous threshold. After reaching the rendezvous threshold, latency for data transfer operations appears to increase by a factor of ten.<o:p></o:p></span></p>
<p class="xmsonormal" style="margin:0in;margin-bottom:.0001pt;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:12.0pt;color:black"> <o:p></o:p></span></p>
<p class="xmsonormal" style="margin:0in;margin-bottom:.0001pt;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:12.0pt;color:black">Specifically, this occurs with verbs and rxm when rxm attempts to register a buffer for transfer using space from the local buffer pool. See
<a href="https://github.com/ofiwg/libfabric/blob/master/prov/util/src/util_buf.c#L115">
https://github.com/ofiwg/libfabric/blob/master/prov/util/src/util_buf.c#L115</a></span><span class="xapple-converted-space"><span style="font-size:12.0pt;color:black"> </span></span><span style="font-size:12.0pt;color:black">for the exact point of failure.
(with rxm_buf_reg as the function being used in that pointer)<o:p></o:p></span></p>
<p class="xmsonormal" style="margin:0in;margin-bottom:.0001pt;font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:12.0pt;color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">--<o:p></o:p></span></p>
<p class="MsoNormal">James Swaro<o:p></o:p></p>
<p class="MsoNormal">Cray Inc.<o:p></o:p></p>
</div>
</body>
</html>