[ofiwg] [libfabric-users] mmap'ed kernel memory in fi_mr_reg

[Jörn Schumacher]

On 11/15/2018 05:37 PM, Hefty, Sean wrote:
>> However, the verbs extensions in the header verbs_exp.h provide
>> additional functionality. Among other things this includes the
>> capability to register a physical address as MR instead of a virtual
>> one. For Mellanox drivers this is described in [1]. The call is
>> ibv_exp_reg_mr with the IBV_EXP_ACCESS_PHYSICAL_ADDR flag.
>>
>> Using the physical address for the reg_mr call we can then transfer
>> data directly from our custom PCIe card to the network adapter,
>> without any copies. Pretty nice!
>>
>> To use the ibv_exp_reg_mr call I had to slightly patch libfabric, see
>> [2]. Libfabric already uses the ibv_exp_reg_mr call if the verbs_exp.h
>> is available, so just the flag needs to be added.
>>
>> As for my feature request, can this be added as official feature to
>> libfabric? I am not sure how many other providers could actually
>> support physical address memory registration, but it is a nice feature
>> when working with custom hardware and drivers. My patch is of course a
>> bit "crude" and this would need to be properly implemented.
>>
>> Thanks.
>>
>> Cheers,
>> Jörn
>>
>>
>> [1] https://community.mellanox.com/docs/DOC-2480
>> [2]
>> https://github.com/joerns/libfabric/compare/v1.6.x...joerns:phys_addr_
>> mr
> 
> Looking at the changes, I see a couple of options.  We can add this as an extension for verbs only.  This could make use of the provider specific flag bits passed into the mr_reg call.  Or, we can define this as a generic mr_reg flag.
> 
> I'm hesitant to make this a more integrated feature, like defining it as a capability bit, since it's completely non-standard and experimental.
> 
> Is there a reliable way for the libfabric provider to determine what type of memory this is, such that the flag passed into verbs can be set automatically as needed?

I am happy with this being a provider-specific extension for now. The 
project that requires this will anyway only use the verbs provider.

I do not think there is a fully reliable way of distinguishing 
physical/virtual addresses. I spoke with our driver expert here who 
cannot think of a way either. The provider-specific flag seems to be the 
way to go, what do you think?

Cheers,
Jörn