[libfabric-users] verbs provider scatter-gather

[smoore at MRU.MEDICAL.CANON]

Hi Sean,

Thanks for the quick response.  Are you saying iovecs are potentially too granular to be turned into SGLs?  For example, a user could pass an iovec with many many entries, which would be difficult for HW to lock in place for transfer.

Can verbs SGLs be accessed via the libfabric API?  My use case is fairly simple.  I have a large data buffer that I want to prefix with a couple hundred bytes of metadata.  My sender passes an iovec with entries for the metadata and the data to fi_sendv.  My receiver passes a similar iovec to fi_recvv.  Is this possible via libfabric, or should I use libibverbs directly?

Best Regards,

Steve

-----Original Message-----
From: Hefty, Sean <sean.hefty at intel.com> 
Sent: Wednesday, December 16, 2020 19:15
To: Moore, Steve <smoore at MRU.MEDICAL.CANON>; libfabric-users at lists.openfabrics.org
Subject: RE: verbs provider scatter-gather

> vrb_mr_regv returns FI_EINVAL if count (the number of entries in the 
> iovec) is larger than 1:
> 
> 
> 
>                 if (OFI_UNLIKELY(count > 1))
> 
>                                 return -FI_EINVAL;
> 
> 
> 
> To the best of my knowledge, libibverbs supports scatter-gather, so 
> why is this check present?  Did I miss some configuration that redefines the OFI_ UNLIKELY macro?

Verbs does not support registering a SGL.  It can used SGLs for send/receive operations, but can only register a single iovec.  For reference, this is the verbs registration call:

struct ibv_mr *ibv_reg_mr(struct ibv_pd *pd, void *addr, size_t length,
			  int access);

Registration of an iovec would likely be difficult for HW to support without imposing some restrictions (e.g. each SGE must be aligned and a multiple of a page size).

- Sean

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com ______________________________________________________________________

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________