[libfabric-users] verbs provider scatter-gather

Fri Dec 18 14:57:05 PST 2020

Hi Sean,

I'm using the verbs FI_EP_MSG endpoint type.  Based on the following in fi_verbs.7.md, I don't think using the memory registration cache / not setting the FI_MR_LOCAL mr_mode bit will work:

Verbs provider requires applications to support the following modes:

FI_EP_MSG endpoint type
    - FI_LOCAL_MR / FI_MR_LOCAL mr mode.

Or am I missing something?

So I think I must register a memory regions for each entry in the iovec by calling fi_mr_reg, and passing the memory regions to fi_recvv.  Or I could call libibverbs ibv_reg_mr, construct a WR and SGL, and call ibv_post_recv.  But that discards a big advantage of libfabric, namely abstracting the providers.  Am I on the right track here?

Thanks again,


-----Original Message-----
From: Hefty, Sean <sean.hefty at intel.com> 
Sent: Thursday, December 17, 2020 16:02
To: Moore, Steve <smoore at MRU.MEDICAL.CANON>; libfabric-users at lists.openfabrics.org
Subject: RE: verbs provider scatter-gather

> Thanks for the quick response.  Are you saying iovecs are potentially 
> too granular to be turned into SGLs?  For example, a user could pass 
> an iovec with many many entries, which would be difficult for HW to lock in place for transfer.

Memory registration is basically an operation that occurs on physical pages.  When you register a region, a page array is created, along with an offset into the first page where the region starts.  When an operation references the region, the HW calculates the offset into the region, and then accesses the page array starting at that offset.

Trying to define a region using an iovec complicates this logic considerably.

Note this is a restriction only for memory registration.  You can freely use SGLs with send/recv operations.

> Can verbs SGLs be accessed via the libfabric API?  My use case is 
> fairly simple.  I have a large data buffer that I want to prefix with a couple hundred bytes of metadata.
> My sender passes an iovec with entries for the metadata and the data 
> to fi_sendv.  My receiver passes a similar iovec to fi_recvv.  Is this 
> possible via libfabric, or should I use libibverbs directly?

You can do this with libfabric.  Actually, some of the libfabric providers do this internally.  What is done internally, is that the metadata (protocol headers in our case) are allocated as a pool and registered.  When passing the data through, a header is retrieved, inserted at the start of an SGL, along with its registration data, and passed through.

If the application above has registered the data buffers, this should be sufficient.  If not, then you would need to register each iovec separately prior to passing the SGL down.

If the buffers are not registered, you can also consider asking libfabric to handle the registration for you by enabling the memory registration cache, or by indicating that you do not support registration by not setting the FI_MR_LOCAL mr_mode bit.

- Sean

This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com ______________________________________________________________________

This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com

More information about the Libfabric-users mailing list