<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;}
@font-face
{font-family:"\@DengXian";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Helvetica Neue";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Menlo;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#467886;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
font-size:10.0pt;
font-family:"Aptos",sans-serif;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
font-size:8.5pt;
font-family:Menlo;
color:black;}
span.s1
{mso-style-name:s1;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:914586943;
mso-list-template-ids:1546566766;}
@list l0:level1
{mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1
{mso-list-id:952320130;
mso-list-template-ids:301891350;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">That’s right. The RDM support in early versions of the verbs provider was separated to become the rxm provider. Today, verbs+ofi_rxm is the preferred choice for RDMA.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Niyaz Murshed <Niyaz.Murshed@arm.com>
<br>
<b>Sent:</b> Tuesday, June 25, 2024 4:08 PM<br>
<b>To:</b> Xiong, Jianxin <jianxin.xiong@intel.com>; libfabric-users@lists.openfabrics.org<br>
<b>Cc:</b> nd <nd@arm.com><br>
<b>Subject:</b> Re: questions on erbs provider for RDMA<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks a lot. I can see the RRoCE packets now with rxm.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">In </span><a href="https://ofiwg.github.io/libfabric/v1.6.1/man/fi_verbs.7.html">fi_verbs(7) (ofiwg.github.io)</a>,
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;background:white">New change in libfabric v1.6: FI_EP_RDM is supported through the OFI RxM utility provider. This is done automatically when the app requests
FI_EP_RDM endpoint. Please refer the man page for RxM provider to learn more. The provider’s internal support for RDM endpoints is deprecated and would be removed from libfabric v1.7 onwards.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;background:white"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;background:white">Does this mean that RDM will not be available with Verbs anymore ?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;background:white">Can we assume that RXM is now the preferred method for RDMA applications?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;background:white"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;background:white">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333;background:white">Niyaz<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div id="mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Xiong, Jianxin <<a href="mailto:jianxin.xiong@intel.com">jianxin.xiong@intel.com</a>><br>
<b>Date: </b>Tuesday, June 25, 2024 at 5:55</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="font-size:12.0pt;color:black">PM<br>
<b>To: </b>Niyaz Murshed <<a href="mailto:Niyaz.Murshed@arm.com">Niyaz.Murshed@arm.com</a>>,
<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a> <<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a>><br>
<b>Cc: </b>nd <<a href="mailto:nd@arm.com">nd@arm.com</a>><br>
<b>Subject: </b>RE: questions on erbs provider for RDMA<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">You explicitly asked for dgram (the domain name ‘mlx5_1-dgram’ only support FI_EP_DGRAM). Use “-d mlx5_1” instead to enable rxm.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Niyaz Murshed <<a href="mailto:Niyaz.Murshed@arm.com">Niyaz.Murshed@arm.com</a>>
<br>
<b>Sent:</b> Tuesday, June 25, 2024 3:22 PM<br>
<b>To:</b> Xiong, Jianxin <<a href="mailto:jianxin.xiong@intel.com">jianxin.xiong@intel.com</a>>;
<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a><br>
<b>Cc:</b> nd <<a href="mailto:nd@arm.com">nd@arm.com</a>><br>
<b>Subject:</b> Re: questions on erbs provider for RDMA</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thank you <span style="color:black;background:white">
Jianxin for the reply. Really appreciate it.</span></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black;background:white"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black;background:white">I ran a test below and took packet capture.</span><o:p></o:p></p>
<p class="p1"><span class="s1">S: fi_rdm_tagged_bw -s </span><span class="apple-converted-space">
</span><span class="s1">192.168.1.100 -d mlx5_1-dgram -I 1 -S 1024</span><o:p></o:p></p>
<p class="p1"><span class="s1">C: fi_rdm_tagged_bw -s </span><span class="apple-converted-space">
</span><span class="s1">192.168.1.200 </span><span class="apple-converted-space">
</span><span class="s1">192.168.1.100 -d</span><span class="apple-converted-space">
</span><span class="s1">mlx5_3-dgram</span><span class="apple-converted-space"> </span>
<span class="s1">-I 1 -S 1024</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The protocol I see is RoCE (not RRoCE) .. Is this expected?pcap attached.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Also.. there is no connection messages anymore, just Send Only. Looking at
<a href="https://www.youtube.com/watch?v=8Cp2KBS4Q4g&t=412s">https://www.youtube.com/watch?v=8Cp2KBS4Q4g&t=412s</a> , am assuming that is expected? All communications are done over “send” ?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">In some servers, I don’t see ofi_rxm, only ofi_rxd .. is there any additional dependencies required for rxm ?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div id="mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Xiong, Jianxin <<a href="mailto:jianxin.xiong@intel.com">jianxin.xiong@intel.com</a>><br>
<b>Date: </b>Tuesday, June 25, 2024 at 4:48</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="font-size:12.0pt;color:black">PM<br>
<b>To: </b>Niyaz Murshed <<a href="mailto:Niyaz.Murshed@arm.com">Niyaz.Murshed@arm.com</a>>,
<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a> <<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a>><br>
<b>Cc: </b>nd <<a href="mailto:nd@arm.com">nd@arm.com</a>><br>
<b>Subject: </b>RE: questions on erbs provider for RDMA</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">All three have Verbs underneath and work over IB or RoCE network. Which provider to use depends on the OFI endpoint type requested by the application.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The verbs provider supports FI_EP_MSG and FI_EP_DGRAM. FI_EP_MSG maps directly to Verbs RC. Each endpoint maps to an RC QP. Explicit connection setup is needed before communication can happen. Multiple endpoints
are needed in order to talk to different peers. FI_EP_DGRAM maps directly to Verbs UD. It’s connectionless, but is unreliable, and message size is limited by MTU size.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Ofi_rxm is a utility provider that runs on top of the verbs provider (using FI_EP_MSG type) and provide connectionless semantics (FI_EP_RDM). Under the cover, each ofi_rxm endpoint maps to verb endpoints and
connection is established automatically on demand. You don’t need to ask for the combination explicitly. When the application asks for “FI_EP_RDM” endpoint type, “verbs;ofi_rxm” is selected automatically.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Ofi_rxd is a utility provider that runs on top the verbs provider (using FI_EP_DGRAM type) and present FI_EP_RDM support. Its functionality is limited compared to ofi_rxm so it is usually not the first choice.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">In summary, an application only needs to ask for the “verbs” provider for RDMA. If an application wants to manage the connection setup by itself, it can ask for ep_type FI_EP_MSG and get the “bare” verbs provider.
The application is then responsible to setup up the connections between endpoints by calling fi_passive_ep(), fi_listen(), fi_connect(), and fi_accept(). If an application doesn’t want to manage connection setup, it can ask for ep_type FI_EP_RDM and get the
“verbs;ofi_rxm” provider. The application then needs to obtain the endpoint address with fi_getname(), exchange the addresses with peers using out-of-band mechanism, and insert the addresses into an address vector. Future communication will use addresses from
the address vector as the identifier for the peer.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">-Jianxin</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Libfabric-users <<a href="mailto:libfabric-users-bounces@lists.openfabrics.org">libfabric-users-bounces@lists.openfabrics.org</a>>
<b>On Behalf Of </b>Niyaz Murshed<br>
<b>Sent:</b> Tuesday, June 25, 2024 2:08 PM<br>
<b>To:</b> <a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a><br>
<b>Cc:</b> nd <<a href="mailto:nd@arm.com">nd@arm.com</a>><br>
<b>Subject:</b> [libfabric-users] questions on erbs provider for RDMA</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi all , </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I am trying to understand the verbs provider specially difference between the below:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<ol style="margin-top:0in" start="1" type="1">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo3"><span style="font-size:11.0pt">Verbs</span><o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo3"><span style="font-size:11.0pt">Verbs;ofi_rxd</span><o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo3"><span style="font-size:11.0pt">Verbs;ofi_rxm</span><o:p></o:p></li></ol>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="p1">They seem to work on 3 different protocols<o:p></o:p></p>
<p class="p1"> <o:p></o:p></p>
<p class="p1"> <o:p></o:p></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><b><i><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333">FI_PROTO_RDMA_CM_IB_RC</span></i></b><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in;line-height:15.0pt;background:white">
<span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333">The protocol runs over Infiniband reliable-connected queue pairs, using the RDMA CM protocol for connection establishment.</span><o:p></o:p></p>
<p class="p1"> <o:p></o:p></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><b><i><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333">FI_PROTO_RXM</span></i></b><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in;line-height:15.0pt;background:white">
<span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333">Reliable-datagram protocol implemented over message endpoints. RXM is a libfabric utility component that adds RDM endpoint semantics over MSG endpoint semantics.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:7.5pt;line-height:15.0pt;background:white">
<b><i><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333">FI_PROTO_RXD</span></i></b><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in;line-height:15.0pt;background:white">
<span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333">Reliable-datagram protocol implemented over datagram endpoints. RXD is a libfabric utility component that adds RDM endpoint semantics over DGRAM endpoint semantics.</span><o:p></o:p></p>
<p class="p1"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><span style="font-size:11.0pt;color:black">From my test of
</span><b><i><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333">FI_PROTO_RDMA_CM_IB_RC</span></i></b><b><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333">, I see in wireshark that, its RoCEv2 protocol
when I test application of RDMA.</span></b><o:p></o:p></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><b><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333">What protocol do RXM and RXD use? I see its TCP packets on the wire? Does it mean it uses TCP ?</span></b><o:p></o:p></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><b><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333">Is it possible to use RoCEv2 for RXM and RXD?</span></b><o:p></o:p></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><b><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333">What is the best provider to use for RDMA ?
</span></b><o:p></o:p></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><b><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333"> </span></b><o:p></o:p></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><b><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333">Regards,</span></b><o:p></o:p></p>
<p class="MsoNormal" style="line-height:15.0pt;background:white"><b><span style="font-size:10.5pt;font-family:"Helvetica Neue",serif;color:#333333">Niyaz</span></b><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>