<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Aptos;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:11.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:65882111;
mso-list-type:hybrid;
mso-list-template-ids:93079008 -897429054 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Aptos",sans-serif;
mso-fareast-font-family:Aptos;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">10/01/2024<o:p></o:p></p>
<p class="MsoNormal"><b><u>Participants:<o:p></o:p></u></b></p>
<p class="MsoNormal">Jianxin Xiong (Intel)<o:p></o:p></p>
<p class="MsoNormal">Amir Shehata (ORNL)<o:p></o:p></p>
<p class="MsoNormal">Alex McKinley (Intel)<o:p></o:p></p>
<p class="MsoNormal">Alexia Ingerson (Intel)<o:p></o:p></p>
<p class="MsoNormal">Ben Lynam (Cornelis)<o:p></o:p></p>
<p class="MsoNormal">Howard Pritchard (LANL)<o:p></o:p></p>
<p class="MsoNormal">Jerome Soumagne (Intel)<o:p></o:p></p>
<p class="MsoNormal">John Byrne (HPE)<o:p></o:p></p>
<p class="MsoNormal">Juee Desai (Intel)<o:p></o:p></p>
<p class="MsoNormal">Nathan Hanford<o:p></o:p></p>
<p class="MsoNormal">Nikhil Nanal (Intel)<o:p></o:p></p>
<p class="MsoNormal">Peinan Zhang (Intel)<o:p></o:p></p>
<p class="MsoNormal">Stephen Oost (Intel)<o:p></o:p></p>
<p class="MsoNormal">Steve Welch (HPE)<o:p></o:p></p>
<p class="MsoNormal">Zach Dworkin (Intel)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b><u>Summary:<o:p></o:p></u></b></p>
<p class="MsoNormal">OFI 2.0 beta plan on track. Early goal is 10/19 with a later goal of 10/25. There will be no RC and no branch.<o:p></o:p></p>
<p class="MsoNormal">Discussion on adding fi_inject2 set of calls which would take in the additional descriptor parameter in order to support FI_HMEM memory with inject calls. There was discussion related to what would happen with backwards compatibility –
OMPI already calls fi_inject with FI_HMEM and some providers, like cxi add iface detection and caching to support FI_HMEM through inject APIs. Conclusion was fi_inject2 would not pass back to fi_inject with a NULL descriptor. No objections. New calls targeted
for beta release.<o:p></o:p></p>
<p class="MsoNormal"><b><u>Notes:<o:p></o:p></u></b></p>
<p class="MsoNormal">OFI 2.0 beta plan:<o:p></o:p></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="mso-list:l0 level1 lfo1">Early goal Friday 10/18<o:p></o:p></li><li class="MsoNormal" style="mso-list:l0 level1 lfo1">If more time needed Friday 10/25<o:p></o:p></li><li class="MsoNormal" style="mso-list:l0 level1 lfo1">Procedure similar to alpha – no RC, no branch<o:p></o:p></li></ul>
<p class="MsoNormal">New inject calls<o:p></o:p></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoNormal" style="mso-list:l0 level1 lfo1">Existing inject calls don’t pass in desc for the buffer<o:p></o:p></li><ul style="margin-top:0in" type="circle">
<li class="MsoNormal" style="mso-list:l0 level2 lfo1">fi_inject(ep, buf, len, dest_addr);<o:p></o:p></li><li class="MsoNormal" style="mso-list:l0 level2 lfo1">Assumption is it’s host memory<o:p></o:p></li></ul>
<li class="MsoNormal" style="mso-list:l0 level1 lfo1">Issues:<o:p></o:p></li><ul style="margin-top:0in" type="circle">
<li class="MsoNormal" style="mso-list:l0 level2 lfo1">Difficult to support FI_HMEM (missing iface info, provider needs to detect iface in order to support FI_HMEM)<o:p></o:p></li><li class="MsoNormal" style="mso-list:l0 level2 lfo1">Providers may have to report inject size of 0 when FI_HMEM is enabled<o:p></o:p></li><li class="MsoNormal" style="mso-list:l0 level2 lfo1">Compromise made by the lnx provider (has to report inject size 0 when one peer provider reports inject size of 0)<o:p></o:p></li><li class="MsoNormal" style="mso-list:l0 level2 lfo1">Performance loss for traffic going through the other peer<o:p></o:p></li></ul>
<li class="MsoNormal" style="mso-list:l0 level1 lfo1">Define new set of inject calls that pass in desc for the buffer<o:p></o:p></li><ul style="margin-top:0in" type="circle">
<li class="MsoNormal" style="mso-list:l0 level2 lfo1">fi_inject2(ep, buf, desc, len, dest_addr);<o:p></o:p></li><li class="MsoNormal" style="mso-list:l0 level2 lfo1">fi_injectdata2(ep, buf, desc, len, data, dest_addr);<o:p></o:p></li><li class="MsoNormal" style="mso-list:l0 level2 lfo1">fi_tinject2(ep, buf, desc, len, dest_addr, tag);<o:p></o:p></li><li class="MsoNormal" style="mso-list:l0 level2 lfo1">fi_tinjectdata2(ep, buf, desc, len, dest_addr, tag);<o:p></o:p></li><li class="MsoNormal" style="mso-list:l0 level2 lfo1">fi_inject_write2(ep, buf, desc, len, dest_addr, addr, key);<o:p></o:p></li><li class="MsoNormal" style="mso-list:l0 level2 lfo1">fi_inject_writedata2(ep, buf, desc, len, data, dest_addr, addr, key);<o:p></o:p></li><li class="MsoNormal" style="mso-list:l0 level2 lfo1">fi_inject_atomic2(ep, buf, desc, count, dest_addr, addr, key, datatype, op);<o:p></o:p></li></ul>
<li class="MsoNormal" style="mso-list:l0 level1 lfo1">OMPI would be able to plug into new call<o:p></o:p></li><li class="MsoNormal" style="mso-list:l0 level1 lfo1">Q: fi_inject2 would pass through to fi_inject with NULL descriptor – how do we know it’s NULL because it’s a host buffer or because the middleware isn’t correctly using API<o:p></o:p></li><ul style="margin-top:0in" type="circle">
<li class="MsoNormal" style="mso-list:l0 level2 lfo1">Do we need to define fi_inject call is not valid for FI_HMEM?<o:p></o:p></li><li class="MsoNormal" style="mso-list:l0 level2 lfo1">Do providers need to add iface detection for regular inject?<o:p></o:p></li></ul>
<li class="MsoNormal" style="mso-list:l0 level1 lfo1">OMPI doesn’t register inject buffers - does CXI provider have caching to support mapping and iface? Yes, but it takes a performance hit; they would like to move to these calls<o:p></o:p></li></ul>
<p class="MsoNormal">Will try to get fi_inject2 calls into beta release<o:p></o:p></p>
</div>
</body>
</html>