<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;}
@font-face
{font-family:"\@DengXian";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif;mso-ligatures:none">2/4/2025<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:14.0pt;font-family:"Calibri",sans-serif;color:#2E75B5;mso-ligatures:none"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-size:14.0pt;font-family:"Calibri",sans-serif;color:#2E75B5;mso-ligatures:none">Participants<o:p></o:p></span></b></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">Alexia Ingerson (Intel)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">Ben Lynam [Cornelis]<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">Bob Cernohous (Cornelis Networks)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">Charles Shereda [Cornelis]<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">Jerome Soumagne (HPE)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">Jianxin Xiong (Intel)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">Juee Desai (Intel)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">Ken Raffenetti (ANL)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">Peinan Zhang (Intel)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">Sai Sunku (AWS)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">Shi Jin (AWS)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">Stephen Oost (Intel)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">Steve Welch [HPE]<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">Zach Dworkin (Intel)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">Nikhil Nanal (Intel)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">Rajalaxmi Angadi (Intel)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none"> <o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:14.0pt;font-family:"Calibri",sans-serif;color:#2E75B5;mso-ligatures:none">Executive Summary<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">An overview of the rearchture if of the OFI shm provider was presented. There are a few drawbacks of the existing shm provider:
(1) command queue can't hold command data after processing; (2) inject pool has high overhead due to access contension; (3) response queue has to be processed in order; (4) CMA-IPC fallback is difficult to implement. With the new design, send side command
isused to allow receiver to hold command data for later use; inject buffer runs in paraller with command queue to avoid pool management; return queue allows out-of-order processing; and CMA-IPC fallback implementation becomes simple because of the reusable
command.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">There was a brief discussion on the rx_size setting in fabtests. The intention was to limit the size of posted receive buffer
to not exceed provider limitation (as defined by max_msg_size). The outcome was that it would be simpler / better to let the provider / driver to allow larger receive buffer be posted. The limit really only needs to be applied to the tx side.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none"> <o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:14.0pt;font-family:"Calibri",sans-serif;color:#2E75B5;mso-ligatures:none">Details<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">OFI shm provider rearchitecure overview:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none"><< See attached slides >><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:27.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-ligatures:none">The discussion about rx_size in fabtests origined from this PR:
<a href="https://github.com/ofiwg/libfabric/pull/10720"><span style="color:blue">https://github.com/ofiwg/libfabric/pull/10720</span></a>. The initial goal was to limit the size of posted buffer based on the max_msg_size setting. However, the situation becomes
more complicated because the same size affect both messaging and RMA buffer and the intention is only to limit the recv buffer not RMA. The true reason behind the original PR was that on some hardware, send/recv size are limited to MTU size, but RMA size is
much larger. During the discussion, it was realized that there is no real reason to not allow larger buffer being posted for recv, it is sufficient to only apply the limit to the sender side. The issue can be resolved more easily by minor change to the provider
/ driver. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
</body>
</html>