<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;}
@font-face
{font-family:"\@DengXian";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Helvetica Neue";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Menlo;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
font-size:8.5pt;
font-family:Menlo;
color:black;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:126319880;
mso-list-type:hybrid;
mso-list-template-ids:-86596980 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1
{mso-list-id:127093124;
mso-list-template-ids:508044226;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">All three have Verbs underneath and work over IB or RoCE network. Which provider to use depends on the OFI endpoint type requested by the application.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The verbs provider supports FI_EP_MSG and FI_EP_DGRAM. FI_EP_MSG maps directly to Verbs RC. Each endpoint maps to an RC QP. Explicit connection setup is needed before communication can happen. Multiple endpoints are needed in order to talk
to different peers. FI_EP_DGRAM maps directly to Verbs UD. It’s connectionless, but is unreliable, and message size is limited by MTU size.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Ofi_rxm is a utility provider that runs on top of the verbs provider (using FI_EP_MSG type) and provide connectionless semantics (FI_EP_RDM). Under the cover, each ofi_rxm endpoint maps to verb endpoints and connection is established automatically
on demand. You don’t need to ask for the combination explicitly. When the application asks for “FI_EP_RDM” endpoint type, “verbs;ofi_rxm” is selected automatically.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Ofi_rxd is a utility provider that runs on top the verbs provider (using FI_EP_DGRAM type) and present FI_EP_RDM support. Its functionality is limited compared to ofi_rxm so it is usually not the first choice.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In summary, an application only needs to ask for the “verbs” provider for RDMA. If an application wants to manage the connection setup by itself, it can ask for ep_type FI_EP_MSG and get the “bare” verbs provider. The application is then
responsible to setup up the connections between endpoints by calling fi_passive_ep(), fi_listen(), fi_connect(), and fi_accept(). If an application doesn’t want to manage connection setup, it can ask for ep_type FI_EP_RDM and get the “verbs;ofi_rxm” provider.
The application then needs to obtain the endpoint address with fi_getname(), exchange the addresses with peers using out-of-band mechanism, and insert the addresses into an address vector. Future communication will use addresses from the address vector as
the identifier for the peer.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-Jianxin<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:"Calibri",sans-serif;mso-ligatures:none">From:</span></b><span style="font-family:"Calibri",sans-serif;mso-ligatures:none"> Libfabric-users <libfabric-users-bounces@lists.openfabrics.org>
<b>On Behalf Of </b>Niyaz Murshed<br>
<b>Sent:</b> Tuesday, June 25, 2024 2:08 PM<br>
<b>To:</b> libfabric-users@lists.openfabrics.org<br>
<b>Cc:</b> nd <nd@arm.com><br>
<b>Subject:</b> [libfabric-users] questions on erbs provider for RDMA<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi all , <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am trying to understand the verbs provider specially difference between the below:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ol style="margin-top:0in" start="1" type="1">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo3">Verbs<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo3">Verbs;ofi_rxd<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo3">Verbs;ofi_rxm<o:p></o:p></li></ol>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="p1">They seem to work on 3 different protocols<o:p></o:p></p>
<p class="p1"><o:p> </o:p></p>
<p class="p1"><o:p> </o:p></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><b><i><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;mso-ligatures:none">FI_PROTO_RDMA_CM_IB_RC</span></i></b><b><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;mso-ligatures:none"><o:p></o:p></span></b></p>
<p class="MsoNormal" style="margin-left:.5in;line-height:15.0pt;background:white">
<span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;mso-ligatures:none">The protocol runs over Infiniband reliable-connected queue pairs, using the RDMA CM protocol for connection establishment.<o:p></o:p></span></p>
<p class="p1"><o:p> </o:p></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><b><i><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;mso-ligatures:none">FI_PROTO_RXM</span></i></b><b><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;mso-ligatures:none"><o:p></o:p></span></b></p>
<p class="MsoNormal" style="margin-left:.5in;line-height:15.0pt;background:white">
<span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;mso-ligatures:none">Reliable-datagram protocol implemented over message endpoints. RXM is a libfabric utility component that adds RDM endpoint semantics over MSG endpoint semantics.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:7.5pt;line-height:15.0pt;background:white">
<b><i><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;mso-ligatures:none">FI_PROTO_RXD</span></i></b><b><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;mso-ligatures:none"><o:p></o:p></span></b></p>
<p class="MsoNormal" style="margin-left:.5in;line-height:15.0pt;background:white">
<span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;mso-ligatures:none">Reliable-datagram protocol implemented over datagram endpoints. RXD is a libfabric utility component that adds RDM endpoint semantics over DGRAM endpoint semantics.<o:p></o:p></span></p>
<p class="p1"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><span style="color:black">From my test of
</span><b><i><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;mso-ligatures:none">FI_PROTO_RDMA_CM_IB_RC</span></i></b><b><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;mso-ligatures:none">, I see
in wireshark that, its RoCEv2 protocol when I test application of RDMA.<o:p></o:p></span></b></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><b><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;mso-ligatures:none">What protocol do RXM and RXD use? I see its TCP packets on the wire? Does it mean it uses
TCP ?<o:p></o:p></span></b></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><b><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;mso-ligatures:none">Is it possible to use RoCEv2 for RXM and RXD?<o:p></o:p></span></b></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><b><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;mso-ligatures:none">What is the best provider to use for RDMA ?
<o:p></o:p></span></b></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><b><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;mso-ligatures:none"><o:p> </o:p></span></b></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><b><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;mso-ligatures:none">Regards,<o:p></o:p></span></b></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><b><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;mso-ligatures:none">Niyaz<o:p></o:p></span></b></p>
</div>
</div>
</body>
</html>