[libfabric-users] mmap'ed kernel memory in fi_mr_reg

[Hefty, Sean]

> I am trying to register a special memory region using fi_mr_reg. This
> is with the verbs provider and libfabric 1.6.2:
> 
> >  fi_mr_reg(socket->domain, buf->data, buf->size, FI_SEND, 0,
> > socket->req_key++, 0, &buf->mr, NULL)
> 
> buf->data is a virtual address that has been mmap'ed from a kernel
> address in a custom kernel driver. The mapping looks uses
> remap_pfn_range:
> 
> >   vma->vm_flags |= VM_DONTEXPAND;
> >   vma->vm_flags |= VM_DONTDUMP;
> >   vma->vm_flags |= VM_LOCKED;
> >   remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff, size, vma-
> >vm_page_prot)
> 
> The call to fi_mr_reg fails with -22 (Invalid argument). If I replace
> the buffer with another buffer allocated simply with malloc, the call
> succeeds.
> 
> Does anybody know why this would not work with mmap'ed memory? Is
> there
> a way of mmap'ing the kernel address to user space that would allow
> the
> memory registration?

I don't know without digging into the kernel.  The OFI call in this case maps directly to the verbs call.  You could try posting this to the linux-rdma mailing list asking about registering mmapped memory.

- Sean