[ofa-general] Problem with libibverbs and huge pages registration.

[Gleb Natapov]

Hi Roland,

   ibv_reg_mr() fails if I try to register a memory region backed by a
huge page, but is not aligned to huge page boundary. Digging deeper I
see that libibverbs aligns memory region to a regular page size and
calls madvise() and the call fails. See program below to reproduce.
The program assumes that hugetlbfs is mounted on /huge and there is at
least one huge page available. I am not use it is possible to know if a
memory buffer is backed by huge page to solve the problem.

Another issue with libibverbs is that after first ibv_reg_mr() fails the
second registration attempt of the same buffer succeed since
ibv_madvise_range() doesn't cleanup after madvice failure and thinks
that memory is already "madvised".

#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>
#include <infiniband/verbs.h>


int main()
{
	int num_devs, fd;
	struct ibv_device **ib_devs;
	struct ibv_context *ctx;
	struct ibv_pd *pd;
	struct ibv_mr *mr;
	char *ptr;
	size_t len = 1024*1024;

	ibv_fork_init();

	ib_devs = ibv_get_device_list(&num_devs);
	ctx = ibv_open_device(ib_devs[0]);
	pd = ibv_alloc_pd(ctx);

	fd = open("/huge/test", O_CREAT | O_RDWR);
	remove("/huge/test");

	ptr = mmap(0, 2*len, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);

	mr = ibv_reg_mr(pd, ptr, len, IBV_ACCESS_LOCAL_WRITE |
					IBV_ACCESS_REMOTE_WRITE |
					IBV_ACCESS_REMOTE_READ);	
	fprintf(stderr, "mr = %p\n", mr);

	return 0;
}
--
			Gleb.