[libfabric-users] mmap'ed kernel memory in fi_mr_reg
joern.schumacher at cern.ch
Fri Nov 16 03:23:57 PST 2018
On 11/15/2018 05:37 PM, Hefty, Sean wrote:
>> However, the verbs extensions in the header verbs_exp.h provide
>> additional functionality. Among other things this includes the
>> capability to register a physical address as MR instead of a virtual
>> one. For Mellanox drivers this is described in . The call is
>> ibv_exp_reg_mr with the IBV_EXP_ACCESS_PHYSICAL_ADDR flag.
>> Using the physical address for the reg_mr call we can then transfer
>> data directly from our custom PCIe card to the network adapter,
>> without any copies. Pretty nice!
>> To use the ibv_exp_reg_mr call I had to slightly patch libfabric, see
>> . Libfabric already uses the ibv_exp_reg_mr call if the verbs_exp.h
>> is available, so just the flag needs to be added.
>> As for my feature request, can this be added as official feature to
>> libfabric? I am not sure how many other providers could actually
>> support physical address memory registration, but it is a nice feature
>> when working with custom hardware and drivers. My patch is of course a
>> bit "crude" and this would need to be properly implemented.
>>  https://community.mellanox.com/docs/DOC-2480
> Looking at the changes, I see a couple of options. We can add this as an extension for verbs only. This could make use of the provider specific flag bits passed into the mr_reg call. Or, we can define this as a generic mr_reg flag.
> I'm hesitant to make this a more integrated feature, like defining it as a capability bit, since it's completely non-standard and experimental.
> Is there a reliable way for the libfabric provider to determine what type of memory this is, such that the flag passed into verbs can be set automatically as needed?
I am happy with this being a provider-specific extension for now. The
project that requires this will anyway only use the verbs provider.
I do not think there is a fully reliable way of distinguishing
physical/virtual addresses. I spoke with our driver expert here who
cannot think of a way either. The provider-specific flag seems to be the
way to go, what do you think?
More information about the Libfabric-users