[ofiwg] [libfabric-users] mmap'ed kernel memory in fi_mr_reg

[Jason Gunthorpe]

On Mon, Nov 26, 2018 at 04:42:49PM +0100, Jörn Schumacher wrote:
> On 11/19/2018 08:42 PM, Hefty, Sean wrote:
> > > > The only alternative I can think of is to try a normal registration
> > > call, and if that fails, try again using the physical flag.  Would
> > > this work, or does the normal registration call succeed, but produce
> > > an unusable MR?
> > > 
> > > This would not work because of a subtlety of the physical memory
> > > registration. The reason is that actually NULL is passed as address in
> > > the call. Check the github link to my patch in the other E-Mail, there
> > > is a line that replaces the address with NULL.
> > > 
> > > If a user passes an illegal virtual address the call should fail. But
> > > if the libfabric call falls back to the physical address registration,
> > > this would then actually succeed as the address is replaced with NULL.
> > 
> > I looked back at the patches and related documentation.  IMO, the verbs physical memory registration interface is just weird.  There is no association between the actual pages and the region AFAICT.
> 
> Indeed this is a rather strange extension.
> 
> I came across a potential solution to adjust our driver to produce memory
> that is compatible with the RDMA stack in the kernel. Supposedly there is an
> alternative to remap_pfn_range. In that case we would not need the physical
> memory registration in libfabric anymore and the overall solution would be
> cleaner (not dependent on the verbs provider)

Several other people have been interested in this, I think many would
appreciate it if you share your solution to the linux-rdma mailing
list.

Jason