[ofa-general] ibv_reg_mr() failing to an mmap'ed iomemory region.

[Peter W. Morreale]

Hi all,

I have an RDMA application that is failing in ibv_reg_mr() and I'm
unsure why.  

The application consists of two parts, a userspace application that is
performing the RDMA transfers via verbs, and a kernel module that
maintains the memory space used in the transfers. 

This is on a SUSE 2.6.22 kernel. 

The kernel module manages a contiguous region of physical memory and
provides a mmap interface to map the regions to userspace.  The
application dirties the memory and periodically transfers the contents
to a remote node.  

The transfer algorithm supports multiple transports including Ethernet,
SDP, and RDMA.  Or rather, is designed to support RDMA. :-)  

Things work fine for Ethernet sockets and SDP, however I consistently
fail attempting to ibv_reg_mr().  Since this is a boolean operation
(returns a pointer or NULL) I'm not sure what is wrong.  

Note that I can bypass the "allocation" portion of the app and
substitute a malloc() instead of asking for the memory region in the
kernel module.  In this case, RDMA (e.g: ibv_reg_mr()) works
flawlessly.  

Since the other transports (and application) can reference the kernel
memory region without issue, I'm a little lost as to what is preventing
ibv_reg_mr() from accessing this space.  

Note that this memory is marked IORESOURCE_BUSY | IORESOURCE_IO.  Does
the kernel-side of ibv_reg_mr() (assuming there is one) do a
__request_region()?  

Thanks for any and all comments...

-PWM