<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#467886;
text-decoration:underline;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
font-size:8.5pt;
font-family:Menlo;
color:black;}
span.s1
{mso-style-name:s1;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:11.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hello, <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Can some please share some knowledge on this <span style="font-family:"Apple Color Emoji"">
😊</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Regards,<o:p></o:p></p>
<p class="MsoNormal">Niyaz<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div id="mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Libfabric-users <libfabric-users-bounces@lists.openfabrics.org> on behalf of Niyaz Murshed <Niyaz.Murshed@arm.com><br>
<b>Date: </b>Tuesday, April 30, 2024 at 10:16</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="font-size:12.0pt;color:black">AM<br>
<b>To: </b>libfabric-users@lists.openfabrics.org <libfabric-users@lists.openfabrics.org><br>
<b>Subject: </b>[libfabric-users] Libfabric questions<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Hello all ,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I am trying to learn libfabric and have some basic questions. Please point me in the right direction.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I am running on ubuntu 22.04 with Mellanox ConnectX 7 NIC cards on an Arm platform (N1).<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><u>Question 1:</u><o:p></o:p></p>
<p class="MsoNormal">Is fi_mlx provider only available on x86?<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Steps : <o:p></o:p></p>
<p class="MsoNormal">Installed UCX (as required in fi_mlx. <a href="https://ofiwg.github.io/libfabric/v1.8.0/man/fi_mlx.7.html">
https://ofiwg.github.io/libfabric/v1.8.0/man/fi_mlx.7.html</a>)<o:p></o:p></p>
<p class="p1"><span class="s1"># Library version: 1.18.0</span><o:p></o:p></p>
<p class="MsoNormal">Install libfabric<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">As per <a href="https://www.intel.com/content/www/us/en/developer/articles/technical/mpi-library-2019-over-libfabric.html">
https://www.intel.com/content/www/us/en/developer/articles/technical/mpi-library-2019-over-libfabric.html</a> this is only available for infiniband devices.<o:p></o:p></p>
<p class="MsoNormal">However, in one of the presentations, it shows mlx also supports RoCE (<a href="https://ibb.co/bPhm0dp">https://ibb.co/bPhm0dp</a>)<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Could you please confirm if MLX provider will work on RoCE ? Other than UCX, do we need to install anything else to enable MLX provider?<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">In my current setup, I only see Verbs provider with doman : mlx<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black">provider: verbs</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> fabric: IB-0xfe80000000000000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> domain: mlx5_0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> version: 121.0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> type: FI_EP_MSG</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> protocol: FI_PROTO_RDMA_CM_IB_RC</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Is Verbs->mlx equivalent to verbs api wrapper over mlx provider?<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><u>Question 2:</u><o:p></o:p></p>
<p class="MsoNormal">Is PSM3 only available for intel NIC cards and only on x86 platform?<o:p></o:p></p>
<p class="MsoNormal">I have the same installation on x86 and Arm , however, I see on x86 platform, the intel NICs has the following enabled<o:p></o:p></p>
<p class="p1"><span class="s1">provider: psm3</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">fabric: TCP-10.118.91.0/24</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">domain: eno1</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">version: 306.10</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">type: FI_EP_RDM</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">protocol: FI_PROTO_PSMX3</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">The same is not present on Arm platform.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><u>Question 3:</u><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">My goal is to run rdma test between 2 application (RoCE).<o:p></o:p></p>
<p class="MsoNormal">I have 2 nics on the server, one on numa0 and another on numa1. I have a loop cable from nic1 to nic2.<o:p></o:p></p>
<p class="MsoNormal">How can I make sure that the data is transferred via the loop cable? Can this be done with fi_rma_bw app?<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Regards,<o:p></o:p></p>
<p class="MsoNormal">Niyaz<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt">IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose
the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
<o:p></o:p></span></p>
</div>
</div>
</div>
</body>
</html>