[libfabric-users] PD/MR sharing by multiple NIC

[Niyaz Murshed]

I assumed PSM3 underneath uses rdma-core  for RoCE protocol.
Slide 8 from https://www.openfabrics.org/wp-content/uploads/2021-workshop-presentations/405_Rimmer_PSM3-Architecture.pdf

So,  PSM3 has its own implementation of RoCE protocol?

From: Xiong, Jianxin <jianxin.xiong at intel.com>
Date: Wednesday, August 21, 2024 at 10:34 AM
To: Niyaz Murshed <Niyaz.Murshed at arm.com>, libfabric-users at lists.openfabrics.org <libfabric-users at lists.openfabrics.org>
Subject: RE: PD/MR sharing by multiple NIC
As I said, you can’t do that with the verbs provider.

From: Niyaz Murshed <Niyaz.Murshed at arm.com>
Sent: Wednesday, August 21, 2024 8:28 AM
To: Xiong, Jianxin <jianxin.xiong at intel.com>; libfabric-users at lists.openfabrics.org
Subject: Re: PD/MR sharing by multiple NIC

Thank you Jianxin.

ibv_alloc_pd takes in ibv_context which comes opening a particular device.
How can we have same PD linked to multiple device ?

Do we have different PDs but same MR?

From: Xiong, Jianxin <jianxin.xiong at intel.com<mailto:jianxin.xiong at intel.com>>
Date: Wednesday, August 21, 2024 at 10:16 AM
To: Niyaz Murshed <Niyaz.Murshed at arm.com<mailto:Niyaz.Murshed at arm.com>>, libfabric-users at lists.openfabrics.org<mailto:libfabric-users at lists.openfabrics.org> <libfabric-users at lists.openfabrics.org<mailto:libfabric-users at lists.openfabrics.org>>
Subject: RE: PD/MR sharing by multiple NIC
It depends on the provider. Since MR is domain level object, in order to share the MR, the endpoints must share the same domain. Some providers (e.g. verbs) map domain to a specific nic and thus can’t share MR among nics. Other providers (e.g. psm3) have domain as pure software entity and can have multiple NICs under the same domain.

Jianxin

From: Libfabric-users <libfabric-users-bounces at lists.openfabrics.org<mailto:libfabric-users-bounces at lists.openfabrics.org>> On Behalf Of Niyaz Murshed
Sent: Wednesday, August 21, 2024 8:00 AM
To: libfabric-users at lists.openfabrics.org<mailto:libfabric-users at lists.openfabrics.org>
Subject: [libfabric-users] PD/MR sharing by multiple NIC

Hello,

Is it possible to multiple NICs share the same PD/MR?
i.e., can I register a memory region that can be used by multiple NICs?

Regards,
Niyaz
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20240821/0025c605/attachment-0001.htm>