[libfabric-users] mmap'ed kernel memory in fi_mr_reg

[Hefty, Sean]

> However, the verbs extensions in the header verbs_exp.h provide
> additional functionality. Among other things this includes the
> capability to register a physical address as MR instead of a virtual
> one. For Mellanox drivers this is described in [1]. The call is
> ibv_exp_reg_mr with the IBV_EXP_ACCESS_PHYSICAL_ADDR flag.
> 
> Using the physical address for the reg_mr call we can then transfer
> data directly from our custom PCIe card to the network adapter,
> without any copies. Pretty nice!
> 
> To use the ibv_exp_reg_mr call I had to slightly patch libfabric, see
> [2]. Libfabric already uses the ibv_exp_reg_mr call if the verbs_exp.h
> is available, so just the flag needs to be added.
> 
> As for my feature request, can this be added as official feature to
> libfabric? I am not sure how many other providers could actually
> support physical address memory registration, but it is a nice feature
> when working with custom hardware and drivers. My patch is of course a
> bit "crude" and this would need to be properly implemented.
> 
> Thanks.
> 
> Cheers,
> Jörn
> 
> 
> [1] https://community.mellanox.com/docs/DOC-2480
> [2]
> https://github.com/joerns/libfabric/compare/v1.6.x...joerns:phys_addr_
> mr

Looking at the changes, I see a couple of options.  We can add this as an extension for verbs only.  This could make use of the provider specific flag bits passed into the mr_reg call.  Or, we can define this as a generic mr_reg flag.

I'm hesitant to make this a more integrated feature, like defining it as a capability bit, since it's completely non-standard and experimental.

Is there a reliable way for the libfabric provider to determine what type of memory this is, such that the flag passed into verbs can be set automatically as needed?

- Sean