[ofiwg] [libfabric-users] mmap'ed kernel memory in fi_mr_reg

[Jason Gunthorpe]

On Thu, Nov 15, 2018 at 04:37:21PM +0000, Hefty, Sean wrote:
> > However, the verbs extensions in the header verbs_exp.h provide
> > additional functionality. Among other things this includes the
> > capability to register a physical address as MR instead of a virtual
> > one. For Mellanox drivers this is described in [1]. The call is
> > ibv_exp_reg_mr with the IBV_EXP_ACCESS_PHYSICAL_ADDR flag.
> > 
> > Using the physical address for the reg_mr call we can then transfer
> > data directly from our custom PCIe card to the network adapter,
> > without any copies. Pretty nice!
> > 
> > To use the ibv_exp_reg_mr call I had to slightly patch libfabric, see
> > [2]. Libfabric already uses the ibv_exp_reg_mr call if the verbs_exp.h
> > is available, so just the flag needs to be added.
> > 
> > As for my feature request, can this be added as official feature to
> > libfabric? I am not sure how many other providers could actually
> > support physical address memory registration, but it is a nice feature
> > when working with custom hardware and drivers. My patch is of course a
> > bit "crude" and this would need to be properly implemented.
> > 
> > Thanks.
> > 
> 
> Looking at the changes, I see a couple of options.  We can add this
> as an extension for verbs only.  This could make use of the provider
> specific flag bits passed into the mr_reg call.  Or, we can define
> this as a generic mr_reg flag.

This is not something I expect we will ever be supporting in upstream
kernel verbs - it simply too insecure.

I don't think libfabric should think about something we in the kernel
would not allow *any* libfabric provider to do.

Jason