<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;}
@font-face
{font-family:"\@DengXian";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#467886;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">That’s right in general, especially w.r.t. verbs.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">However, in a more general case that may not fit your specific scenario, if the provider allows user supplied memory keys, it’s possible to have to two registrations using the same key. Keys only need to be
unique within the same domain, so they can be reused in different domains. <o:p>
</o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Niyaz Murshed <Niyaz.Murshed@arm.com>
<br>
<b>Sent:</b> Wednesday, August 21, 2024 10:07 AM<br>
<b>To:</b> Xiong, Jianxin <jianxin.xiong@intel.com>; libfabric-users@lists.openfabrics.org<br>
<b>Subject:</b> Re: PD/MR sharing by multiple NIC<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks Jianxin. That means the same memory can be shared by multiple domain.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Let’s say we We have 1 large memory region and 2 NICs used by the process on both server and client.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This would mean the remote node should have the knowledge of which NIC it is sending the write out from and should use the appropriate Rkey to use for that Write message.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We cannot have a common Rkey per rank(process) for this region so that remote node doesn’t have to worry which NIC is receiving it.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div id="mail-editor-reference-message-container">
<div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Xiong, Jianxin <<a href="mailto:jianxin.xiong@intel.com">jianxin.xiong@intel.com</a>><br>
<b>Date: </b>Wednesday, August 21, 2024 at 11:51</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="font-size:12.0pt;color:black">AM<br>
<b>To: </b>Niyaz Murshed <<a href="mailto:Niyaz.Murshed@arm.com">Niyaz.Murshed@arm.com</a>>,
<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a> <<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a>><br>
<b>Subject: </b>RE: PD/MR sharing by multiple NIC<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">There is a caching mechanism for the registration.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">In general case (not psm3), you could pre-register the same buffer for each domain, and choose the right mr to use depending on which domain the endpoint being worked belongs to.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Niyaz Murshed <<a href="mailto:Niyaz.Murshed@arm.com">Niyaz.Murshed@arm.com</a>>
<br>
<b>Sent:</b> Wednesday, August 21, 2024 9:42 AM<br>
<b>To:</b> Xiong, Jianxin <<a href="mailto:jianxin.xiong@intel.com">jianxin.xiong@intel.com</a>>;
<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a><br>
<b>Subject:</b> Re: PD/MR sharing by multiple NIC</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thank you Jianxin. Really appreciate the explanation.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">“An MR only needs to register with the NIC when needed.â€â€”</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I was wondering about the cost of registration on every read/write.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I will check detail of “rv†module to see what is happening.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I was thinking to avoid the cost of registration on read/write by pre-registering a large memory during init. However, that doesn’t work if multiple NICs cannot share the same pre-registered memory.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div id="mail-editor-reference-message-container">
<div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Xiong, Jianxin <<a href="mailto:jianxin.xiong@intel.com">jianxin.xiong@intel.com</a>><br>
<b>Date: </b>Wednesday, August 21, 2024 at 11:34</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="font-size:12.0pt;color:black">AM<br>
<b>To: </b>Niyaz Murshed <<a href="mailto:Niyaz.Murshed@arm.com">Niyaz.Murshed@arm.com</a>>,
<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a> <<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a>><br>
<b>Subject: </b>RE: PD/MR sharing by multiple NIC</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">For psm3, domain is purely a software object. It’s the endpoint that maps to a specific nic (and ibv pd). An MR only needs to register with the NIC when needed.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">With RoCE, PSM3 can work over UD or RC. When using UD, user data are always copied over pre-registered internal buffers that is allocated per endpoint. User MR doesn’t need to be registered with the NIC.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">When using RC, registration with the NIC is only needed for RDMA read/write operations. This is done via the optional “rv†kernel module.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Niyaz Murshed <<a href="mailto:Niyaz.Murshed@arm.com">Niyaz.Murshed@arm.com</a>>
<br>
<b>Sent:</b> Wednesday, August 21, 2024 8:54 AM<br>
<b>To:</b> Xiong, Jianxin <<a href="mailto:jianxin.xiong@intel.com">jianxin.xiong@intel.com</a>>;
<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a><br>
<b>Subject:</b> Re: PD/MR sharing by multiple NIC</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I assumed PSM3 underneath uses rdma-core for RoCE protocol.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Slide 8 from <a href="https://www.openfabrics.org/wp-content/uploads/2021-workshop-presentations/405_Rimmer_PSM3-Architecture.pdf">
https://www.openfabrics.org/wp-content/uploads/2021-workshop-presentations/405_Rimmer_PSM3-Architecture.pdf</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">So, PSM3 has its own implementation of RoCE protocol?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div id="mail-editor-reference-message-container">
<div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Xiong, Jianxin <<a href="mailto:jianxin.xiong@intel.com">jianxin.xiong@intel.com</a>><br>
<b>Date: </b>Wednesday, August 21, 2024 at 10:34</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="font-size:12.0pt;color:black">AM<br>
<b>To: </b>Niyaz Murshed <<a href="mailto:Niyaz.Murshed@arm.com">Niyaz.Murshed@arm.com</a>>,
<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a> <<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a>><br>
<b>Subject: </b>RE: PD/MR sharing by multiple NIC</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">As I said, you can’t do that with the verbs provider.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Niyaz Murshed <<a href="mailto:Niyaz.Murshed@arm.com">Niyaz.Murshed@arm.com</a>>
<br>
<b>Sent:</b> Wednesday, August 21, 2024 8:28 AM<br>
<b>To:</b> Xiong, Jianxin <<a href="mailto:jianxin.xiong@intel.com">jianxin.xiong@intel.com</a>>;
<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a><br>
<b>Subject:</b> Re: PD/MR sharing by multiple NIC</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thank you Jianxin.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">ibv_alloc_pd takes in ibv_context which comes opening a particular device.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">How can we have same PD linked to multiple device ?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Do we have different PDs but same MR?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div id="mail-editor-reference-message-container">
<div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Xiong, Jianxin <<a href="mailto:jianxin.xiong@intel.com">jianxin.xiong@intel.com</a>><br>
<b>Date: </b>Wednesday, August 21, 2024 at 10:16</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="font-size:12.0pt;color:black">AM<br>
<b>To: </b>Niyaz Murshed <<a href="mailto:Niyaz.Murshed@arm.com">Niyaz.Murshed@arm.com</a>>,
<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a> <<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a>><br>
<b>Subject: </b>RE: PD/MR sharing by multiple NIC</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">It depends on the provider. Since MR is domain level object, in order to share the MR, the endpoints must share the same domain. Some providers (e.g. verbs) map domain to a specific nic and thus can’t share
MR among nics. Other providers (e.g. psm3) have domain as pure software entity and can have multiple NICs under the same domain.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Jianxin</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Libfabric-users <<a href="mailto:libfabric-users-bounces@lists.openfabrics.org">libfabric-users-bounces@lists.openfabrics.org</a>>
<b>On Behalf Of </b>Niyaz Murshed<br>
<b>Sent:</b> Wednesday, August 21, 2024 8:00 AM<br>
<b>To:</b> <a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a><br>
<b>Subject:</b> [libfabric-users] PD/MR sharing by multiple NIC</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Hello, </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Is it possible to multiple NICs share the same PD/MR?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">i.e., can I register a memory region that can be used by multiple NICs?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Regards,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Niyaz</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:12.0pt">IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose
the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt">IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose
the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt">IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose
the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt">IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose
the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt">IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose
the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
<o:p></o:p></span></p>
</div>
</div>
</body>
</html>