[libfabric-users] mmap'ed kernel memory in fi_mr_reg

[Jörn Schumacher]

Hi all,

I am trying to register a special memory region using fi_mr_reg. This is 
with the verbs provider and libfabric 1.6.2:

>  fi_mr_reg(socket->domain, buf->data, buf->size, FI_SEND, 0, socket->req_key++, 0, &buf->mr, NULL)

buf->data is a virtual address that has been mmap'ed from a kernel 
address in a custom kernel driver. The mapping looks uses remap_pfn_range:

>   vma->vm_flags |= VM_DONTEXPAND;
>   vma->vm_flags |= VM_DONTDUMP;
>   vma->vm_flags |= VM_LOCKED;
>   remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff, size, vma->vm_page_prot)

The call to fi_mr_reg fails with -22 (Invalid argument). If I replace 
the buffer with another buffer allocated simply with malloc, the call 
succeeds.

Does anybody know why this would not work with mmap'ed memory? Is there 
a way of mmap'ing the kernel address to user space that would allow the 
memory registration?

Thanks!

Cheers,
Jörn