[libfabric-users] FI_MR_ENDPOINT question

Biddiscombe, John A. john.biddiscombe at cscs.ch
Thu Sep 1 11:52:39 PDT 2022


Thanks very much, that discussion is very helpful (And yes, when I said send/recv - what I really meant was initiating transfers including RMA on the send endpoint, and pre-posting rx buffers on the recv one)



From: Welch, Steve <welch at hpe.com>
Sent: 01 September 2022 17:05:16
To: Biddiscombe, John A.; libfabric-users at lists.openfabrics.org
Subject: Re: FI_MR_ENDPOINT question

Hi John,

You mention messaging (send/recv) in your question; if FI_MR_LOCAL is not required, then for those operations the registration for local access is transparent to the client and multiple EP within the same domain/VA space could transparently share the underlying DMA mapping.

For RMA (write/read/amo) where remote MR are explicitly registered, bound and enabled, there is a good discussion here:


From: Libfabric-users <libfabric-users-bounces at lists.openfabrics.org> on behalf of Biddiscombe, John A. <john.biddiscombe at cscs.ch>
Date: Thursday, September 1, 2022 at 4:22 AM
To: libfabric-users at lists.openfabrics.org <libfabric-users at lists.openfabrics.org>
Subject: [libfabric-users] FI_MR_ENDPOINT question

For machines that set the FI_MR_ENDPOINT flag - we must bind memory regions to an endpoint.

If I use a separate endpoint for sending and receiving, or worse, threadlocal endpoints (for sending), or a scalable endpoint (for sending)

Must I only use memory regions bound to the exact endpoint I'm actually using for a send  or receive operation (meaning threadlocal pools of registered memory for a threadlocal endpoint setup and separate pools of memory for send/receive).

How strict are the rules? If I bind memory to endpoint 1, and send using endpoint 2 - will it break?


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20220901/2a2faa40/attachment-0001.htm>

More information about the Libfabric-users mailing list