[ofiwg] [libfabric-users] mmap'ed kernel memory in fi_mr_reg
sean.hefty at intel.com
Thu Nov 15 08:37:21 PST 2018
> However, the verbs extensions in the header verbs_exp.h provide
> additional functionality. Among other things this includes the
> capability to register a physical address as MR instead of a virtual
> one. For Mellanox drivers this is described in . The call is
> ibv_exp_reg_mr with the IBV_EXP_ACCESS_PHYSICAL_ADDR flag.
> Using the physical address for the reg_mr call we can then transfer
> data directly from our custom PCIe card to the network adapter,
> without any copies. Pretty nice!
> To use the ibv_exp_reg_mr call I had to slightly patch libfabric, see
> . Libfabric already uses the ibv_exp_reg_mr call if the verbs_exp.h
> is available, so just the flag needs to be added.
> As for my feature request, can this be added as official feature to
> libfabric? I am not sure how many other providers could actually
> support physical address memory registration, but it is a nice feature
> when working with custom hardware and drivers. My patch is of course a
> bit "crude" and this would need to be properly implemented.
>  https://community.mellanox.com/docs/DOC-2480
Looking at the changes, I see a couple of options. We can add this as an extension for verbs only. This could make use of the provider specific flag bits passed into the mr_reg call. Or, we can define this as a generic mr_reg flag.
I'm hesitant to make this a more integrated feature, like defining it as a capability bit, since it's completely non-standard and experimental.
Is there a reliable way for the libfabric provider to determine what type of memory this is, such that the flag passed into verbs can be set automatically as needed?
More information about the ofiwg