<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
@font-face
{font-family:"Apple Color Emoji";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#467886;
text-decoration:underline;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
font-size:8.5pt;
font-family:Menlo;
color:black;}
span.s1
{mso-style-name:s1;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle230
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Thank Jianxin for the reply.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I can see that psm3 is not being enabled on arm platform. I will check on that.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:#32F4F1">/* psm3 provider is built */<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:#63E2FF;background:yellow;mso-highlight:yellow">#define HAVE_PSM3
</span><span style="font-size:10.5pt;font-family:Menlo;color:#FB1EFF;background:yellow;mso-highlight:yellow">0</span><span style="font-size:10.5pt;font-family:Menlo;color:#63E2FF"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:#32F4F1">/* psm3 provider is built as DSO */<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:#63E2FF">#define HAVE_PSM3_DL
</span><span style="font-size:10.5pt;font-family:Menlo;color:#FB1EFF">0</span><span style="font-size:10.5pt;font-family:Menlo;color:#63E2FF"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:#32F4F1">/* PSM3 source is built-in */<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:#63E2FF">#define HAVE_PSM3_SRC
</span><span style="font-size:10.5pt;font-family:Menlo;color:#FB1EFF">1</span><span style="font-size:10.5pt;font-family:Menlo;color:#63E2FF"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">On x86 machine :<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:#32F4F1">/* psm3 provider is built */<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:#63E2FF">#define HAVE_PSM3
</span><span style="font-size:10.5pt;font-family:Menlo;color:#FB1EFF">1</span><span style="font-size:10.5pt;font-family:Menlo;color:#63E2FF"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:#32F4F1">/* psm3 provider is built as DSO */<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:#63E2FF">#define HAVE_PSM3_DL
</span><span style="font-size:10.5pt;font-family:Menlo;color:#FB1EFF">0</span><span style="font-size:10.5pt;font-family:Menlo;color:#63E2FF"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:#32F4F1">/* PSM3 source is built-in */<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:#63E2FF">#define HAVE_PSM3_SRC
</span><span style="font-size:10.5pt;font-family:Menlo;color:#FB1EFF">1</span><span style="font-size:10.5pt;font-family:Menlo;color:#63E2FF"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">However, one question I have is for x86. The machine has both Intel and Mellanox NICs.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black">Root#/libfabric# lshw -c net -businfo -numeric<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black">Bus info Device Class Description<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black">============================================================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"><a href="mailto:pci@0000:01:00.0">pci@0000:01:00.0</a> eno1 network I210 Gigabit Network Connection [8086:1533]<o:p></o:p></span></p>
<p class="p1"><span style="font-size:10.5pt"><a href="mailto:pci@0000:40:00.0">pci@0000:40:00.0</a> ens2f0np0 network MT2910 Family [ConnectX-7] [15B3:1021] (</span><b><span style="font-size:10.5pt;color:#CA3323">Me</span></b><span style="font-size:10.5pt">llanox
Technologies MT2910 Family [ConnectX-7])<o:p></o:p></span></p>
<p class="p1"><span style="font-size:10.5pt"><o:p> </o:p></span></p>
<p class="p1"><span style="font-size:10.5pt">But when I run fi_info, psm3 is only enabled on for the intel nic<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black">root#fi_info -p psm3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black">provider: psm3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> fabric: TCP-10.118.91.0/24<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> domain: eno1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> version: 306.10<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> type: FI_EP_RDM<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> protocol: FI_PROTO_PSMX3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black">provider: psm3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> fabric: TCP-10.118.91.0/24<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> domain: eno1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> version: 306.10<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> type: FI_EP_RDM<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> protocol: FI_PROTO_PSMX3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black">provider: psm3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> fabric: TCP-10.118.91.0/24<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> domain: eno1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> version: 306.10<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> type: FI_EP_RDM<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> protocol: FI_PROTO_PSMX3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black">root # lscpu <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black">Architecture: x86_64<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> CPU op-mode(s): 32-bit, 64-bit<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black">Vendor ID: GenuineIntel<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> Model name: Intel(R) Xeon(R) Platinum 8480+<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Is this expected ? Shouldn’t it show up for the Mellanox devices too?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black">root # ibv_devinfo<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black">hca_id: mlx5_0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> transport: InfiniBand (0)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> fw_ver: 28.39.3004<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> node_guid: 946d:ae03:007d:a230<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> sys_image_guid: 946d:ae03:007d:a230<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> vendor_id: 0x02c9<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> vendor_part_id: 4129<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> hw_ver: 0x0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> board_id: MT_0000000834<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> phys_port_cnt: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> port: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> state: PORT_ACTIVE (4)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> max_mtu: 4096 (5)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> active_mtu: 1024 (3)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> sm_lid: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> port_lid: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> port_lmc: 0x00<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> link_layer: Ethernet<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Niyaz<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div id="mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Xiong, Jianxin <jianxin.xiong@intel.com><br>
<b>Date: </b>Wednesday, May 8, 2024 at 12:33</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="font-size:12.0pt;color:black">AM<br>
<b>To: </b>Niyaz Murshed <Niyaz.Murshed@arm.com>, libfabric-users@lists.openfabrics.org <libfabric-users@lists.openfabrics.org><br>
<b>Cc: </b>nd <nd@arm.com><br>
<b>Subject: </b>RE: Libfabric questions<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Hi Niyaz,</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">The mlx provider is only available as part of Intel MPI package, which is only available for x86.</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">If you build libfabric from source, you can use the ucx provider instead. Similar to the mlx provider, the ucx provider runs on top of UCX.</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">The verbs provider runs on top of IB Verbs. The domain name “mlx5_0” is the device name of Mellanox NIC. It has nothing to do with the mlx provider, nor UCX.</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">The psm3 provider supports Intel NICs as well as Ethernet/IB/RoCE NICs from other vendors. Whether it is present on a machine depends on which package is installed. For example,
if it is installed as part of the Intel Ethernet Fabric Suite package, then it is only available for x86_64. You may try to build it from libfabric source on other platforms, by adding “—enable-psm3” to the libfabric “configure” command line. If configure
fails, check the content of “config.log” to see what is missing. This is an untested territory so I can’t guarantee that it will work.</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">The third question has been answered off-line, so I won’t repeat here.</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">-Jianxin</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Libfabric-users <libfabric-users-bounces@lists.openfabrics.org>
<b>On Behalf Of </b>Niyaz Murshed<br>
<b>Sent:</b> Monday, May 6, 2024 6:16 PM<br>
<b>To:</b> libfabric-users@lists.openfabrics.org<br>
<b>Cc:</b> nd <nd@arm.com><br>
<b>Subject:</b> Re: [libfabric-users] Libfabric questions</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Hello, <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Can some please share some knowledge on this
</span><span style="font-size:11.0pt;font-family:"Apple Color Emoji"">😊</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Niyaz<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<div id="mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Libfabric-users <<a href="mailto:libfabric-users-bounces@lists.openfabrics.org">libfabric-users-bounces@lists.openfabrics.org</a>> on behalf of Niyaz Murshed <<a href="mailto:Niyaz.Murshed@arm.com">Niyaz.Murshed@arm.com</a>><br>
<b>Date: </b>Tuesday, April 30, 2024 at 10:16</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="font-size:12.0pt;color:black">AM<br>
<b>To: </b><a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a> <<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a>><br>
<b>Subject: </b>[libfabric-users] Libfabric questions</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hello all ,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I am trying to learn libfabric and have some basic questions. Please point me in the right direction.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I am running on ubuntu 22.04 with Mellanox ConnectX 7 NIC cards on an Arm platform (N1).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><u><span style="font-size:11.0pt">Question 1:</span></u><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Is fi_mlx provider only available on x86?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Steps : <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Installed UCX (as required in fi_mlx.
<a href="https://ofiwg.github.io/libfabric/v1.8.0/man/fi_mlx.7.html">https://ofiwg.github.io/libfabric/v1.8.0/man/fi_mlx.7.html</a>)<o:p></o:p></span></p>
<p class="p1"><span class="s1"># Library version: 1.18.0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Install libfabric<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">As per <a href="https://www.intel.com/content/www/us/en/developer/articles/technical/mpi-library-2019-over-libfabric.html">
https://www.intel.com/content/www/us/en/developer/articles/technical/mpi-library-2019-over-libfabric.html</a> this is only available for infiniband devices.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">However, in one of the presentations, it shows mlx also supports RoCE (<a href="https://ibb.co/bPhm0dp">https://ibb.co/bPhm0dp</a>)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Could you please confirm if MLX provider will work on RoCE ? Other than UCX, do we need to install anything else to enable MLX provider?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">In my current setup, I only see Verbs provider with doman : mlx<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black">provider: verbs</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> fabric: IB-0xfe80000000000000</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> domain: mlx5_0</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> version: 121.0</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> type: FI_EP_MSG</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black"> protocol: FI_PROTO_RDMA_CM_IB_RC</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Is Verbs->mlx equivalent to verbs api wrapper over mlx provider?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><u><span style="font-size:11.0pt">Question 2:</span></u><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Is PSM3 only available for intel NIC cards and only on x86 platform?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I have the same installation on x86 and Arm , however, I see on x86 platform, the intel NICs has the following enabled<o:p></o:p></span></p>
<p class="p1"><span class="s1">provider: psm3</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">fabric: TCP-10.118.91.0/24</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">domain: eno1</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">version: 306.10</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">type: FI_EP_RDM</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">protocol: FI_PROTO_PSMX3</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The same is not present on Arm platform.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><u><span style="font-size:11.0pt">Question 3:</span></u><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">My goal is to run rdma test between 2 application (RoCE).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I have 2 nics on the server, one on numa0 and another on numa1. I have a loop cable from nic1 to nic2.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">How can I make sure that the data is transferred via the loop cable? Can this be done with fi_rma_bw app?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Niyaz<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt">IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose
the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>