[libfabric-users] verbs provider scatter-gather

[Hefty, Sean]

> Thanks for the quick response.  Are you saying iovecs are potentially too granular to
> be turned into SGLs?  For example, a user could pass an iovec with many many entries,
> which would be difficult for HW to lock in place for transfer.

Memory registration is basically an operation that occurs on physical pages.  When you register a region, a page array is created, along with an offset into the first page where the region starts.  When an operation references the region, the HW calculates the offset into the region, and then accesses the page array starting at that offset.

Trying to define a region using an iovec complicates this logic considerably.

Note this is a restriction only for memory registration.  You can freely use SGLs with send/recv operations.

> Can verbs SGLs be accessed via the libfabric API?  My use case is fairly simple.  I
> have a large data buffer that I want to prefix with a couple hundred bytes of metadata.
> My sender passes an iovec with entries for the metadata and the data to fi_sendv.  My
> receiver passes a similar iovec to fi_recvv.  Is this possible via libfabric, or should
> I use libibverbs directly?

You can do this with libfabric.  Actually, some of the libfabric providers do this internally.  What is done internally, is that the metadata (protocol headers in our case) are allocated as a pool and registered.  When passing the data through, a header is retrieved, inserted at the start of an SGL, along with its registration data, and passed through.

If the application above has registered the data buffers, this should be sufficient.  If not, then you would need to register each iovec separately prior to passing the SGL down.

If the buffers are not registered, you can also consider asking libfabric to handle the registration for you by enabling the memory registration cache, or by indicating that you do not support registration by not setting the FI_MR_LOCAL mr_mode bit.

- Sean