[ofa-general] mmap() and ibv_reg_mr() and RDMA

[Joerg Zinke]

On Tue, 22 May 2007 13:46:04 -0700
Roland Dreier <rdreier at cisco.com> wrote:

>  > I want to do RDMA-write on mmap'ed memory.
>  > but it fails to register the memory region.
>  > 
>  > is there something special, to use ibv_reg_mr() on memory which I
>  > got from mmap()?
>  > it works fine with plain allocated memory (with memalign()).
>  > memory is page-aligned in both cases.
> 
> How exactly are you mmap()ing the memory?  memalign(), malloc() etc
> are implemented with mmap() internally, so obviously memory
> registration of some mmap()ed memory is fine.

i created a character device in the kernel and registered memory with
kzalloc():

if ((kmalloc_ptr = kzalloc((NPAGES + 2) * PAGE_SIZE, GFP_KERNEL |
__GFP_DMA)) == NULL) { return -ENOMEM; }

rounded to page bondary:

kmalloc_area = (struct serverinfo *)((((unsigned long)kmalloc_ptr) +
PAGE_SIZE - 1) & PAGE_MASK);

this area is mapped via the character device and with
the help of remap_pfn_range() into userspace... this works fine i can
access it from userspace and write/read from it:

#define MMAP_AREA_LEN (NPAGES*getpagesize())

...

mmap_area = (struct serverinfo*)mmap(0, MMAP_AREA_LEN,
PROT_READ|PROT_WRITE, MAP_SHARED| MAP_LOCKED, fd, MMAP_AREA_LEN);

but when i try to register the mmap_area with ibv_reg_mr() it
fails. 

regards,

joerg