<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Aptos;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#467886;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;}
span.EmailStyle18
{mso-style-type:personal-compose;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:11.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1659114684;
mso-list-type:hybrid;
mso-list-template-ids:833901650 -984604798 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;
mso-fareast-font-family:Aptos;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">We’re looking at alternative network I/O APIs including libfabric. The objective is to support multiple hardware vendors without sacrificing latency or throughput.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">While working my way through OFI docs, I’ve come up with a couple of basic questions that don’t quite fallout into my lap at least to my eye.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The first issue pertains to basic API roles and responsibilities with respect to application messages, or what OFI seems to call “message boundaries.” For example, APIs like HOMA, eRPC --- granted experimental --- explicitly guarantee message
delivery (at least or exactly once). DPDK is fundamentally packet oriented and, as a practical matter, makes no guarantees about application messages.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The OFI situation --- correct me if wrong --- seems to be,<br>
<br>
<o:p></o:p></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Regardless of the provider, full application messages will be sent and received exactly once provided the endpoint(s) is configured with one of FI_EP_DGRAM, FI_EP_MSG, FI_EP_RDM<o:p></o:p>
<ul style="margin-top:0in" type="circle">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo1">FI_EP_DGRAM has a caveat about maximum message size<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo1"><a href="https://ofiwg.github.io/libfabric/v1.21.0/man/fi_endpoint.3.html">While not explicitly stated</a> the guarantee seems to be whole messages are sent and delivered, but there’s
no guarantee about message order<o:p></o:p></li></ul>
</li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Regardless of the provider, no guarantees are made if the endpoint is configured with FI_EP_DGRAM. To make it reliable in this mode programmers must:<o:p></o:p>
<ul style="margin-top:0in" type="circle">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo1">Define a message reliability policy<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level2 lfo1">Use OFI send/recv calls to implement a protocol whereby senders and receivers exchange information until each side is convinced the policy is satisfied. This will likely leverage the
usual suspects of ACKs, Grants, NACKs, CRC checks and so on<o:p></o:p></li></ul>
</li></ul>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The second issue is multi-cast. Some of our messaging architecture leverages pub-sub through Infiniband multi-cast. Now, when I look at provider details, for example,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1"><a href="https://ofiwg.github.io/libfabric/v1.21.0/man/fi_verbs.7.html">Verbs</a><o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1"><a href="https://ofiwg.github.io/libfabric/v1.21.0/man/fi_udp.7.html">UDP</a><o:p></o:p></li></ul>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I had hoped to see a blurb mentioning whether or not mcast is supported method. Where can I find more details on this?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Finally, OFI’s support for <a href="https://ofiwg.github.io/libfabric/v1.21.0/man/fi_direct.7.html">
Direct</a> caught my eye. Here again provider pages are not tipping their hand as to whether there’s a direct implementation. How can I pursue this detail?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you,<br>
Shane Miller<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message
may not represent the opinion of Intercontinental Exchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards
from viruses and pursue alternate means of communication where privacy or a binding message is desired.<br>
</font>
</body>
</html>