<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style id="mysingle_style">v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
<title>Samsung Enterprise Portal mySingle</title>
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;
mso-believe-normal-left:yes;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
margin-top:3.75pt;
margin-right:0in;
margin-bottom:3.75pt;
margin-left:0in;
font-size:9.0pt;
font-family:"Arial",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><![if mso 9]><style>p.MsoNormal
{margin-left:7.5pt;}
</style><![endif]><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="margin-left:7.5pt;margin-top:7.5pt;margin-right:7.5pt;margin-bottom:7.5pt">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Suman,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">There are two important boundary conditions to consider with analyzing the queue to core and MSI-X mapping. These boundary conditions include systems with more
than 32 cores and also more than 64 cores. On a system with more than 32 cores, in your example below, it’s possible some cores won’t get mapped to the appropriate MSI-X vectors. The other boundary condition is when the system has more than 64 cores. In
this case, the core numbering may not be consecutive. In both cases, without re-creating the queues, there are likely to be problems.<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">All that being said, I do see areas for improvements in terms of memory allocation per NUMA node and the core to queue pair assignment.
Is it possible for you to do more investigation and to come back with a proposal for a fix and potentially a patch?<o:p></o:p></span></a></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Carolyn<o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> nvmewin-bounces@lists.openfabrics.org [mailto:nvmewin-bounces@lists.openfabrics.org]
<b>On Behalf Of </b>SUMAN PRAKASH B<br>
<b>Sent:</b> Monday, June 15, 2015 10:36 AM<br>
<b>To:</b> nvmewin@lists.openfabrics.org<br>
<b>Subject:</b> [nvmewin] regarding incorrect core-queue-msix mapping in 1.4 OFA code<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p>Dear All,<o:p></o:p></p>
<p> <o:p></o:p></p>
<p>The core-queue-msix mapping(NUMA implementation) in the 1.4 OFA code doesn't look correct.<o:p></o:p></p>
<p> <o:p></o:p></p>
<p>During learning-cores, after the driver sends a read command on each queue to get the Core to MSIx mapping, the driver updates the following:<o:p></o:p></p>
<p>a) MSIx vector corresponding to core. <o:p></o:p></p>
<p>b) Queue assignment with corresponding core.<o:p></o:p></p>
<p>Because of b, the queue assignment is changed but the memory allocation remains unchanged. Then when the queues are deleted and re-created, we see that the queues are created on memory from other nodes, which should not be the case, as this will introduce
remote node memory access which might impact the performance.<o:p></o:p></p>
<p> <o:p></o:p></p>
<p>We tested on a 32 logical processor server with 2 NUMA nodes and following is the mapping after learning cores -
<o:p></o:p></p>
<p> <o:p></o:p></p>
<p><img width="330" height="666" id="_x0000_i1025" src="cid:image001.gif@01D0A785.DFC282E0"><o:p></o:p></p>
<p> <o:p></o:p></p>
<p>Ideally, during learning-cores, after the driver sends a read command on each queue to get the Core to MSIx mapping, the driver should update only the following:<o:p></o:p></p>
<p>a) MSIx vector corresponding to core. <o:p></o:p></p>
<p>With only a), following is the core-queue-msix mapping:<o:p></o:p></p>
<p> <o:p></o:p></p>
<p><img width="325" height="665" id="_x0000_i1026" src="cid:image002.gif@01D0A785.DFC282E0"> <o:p></o:p></p>
<p> <o:p></o:p></p>
<p><strong><span style="font-family:"Arial",sans-serif">Any comments?</span></strong><o:p></o:p></p>
<p> <o:p></o:p></p>
<p>Thanks,<o:p></o:p></p>
<p>Suman<o:p></o:p></p>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><span style="border:solid windowtext 1.0pt;padding:0in"><img width="100" height="100" id="_x0000_i1027" src="cid:~WRD000.jpg" alt="Image removed by sender."></span><o:p></o:p></p>
</div>
</body>
</html>