[openib-general] IBV_WC_LOC_PROT_ERR on RDMA read into huge paged buffers acquiered with shmget....

[Bub Thomas]

Our application uses huge paged (shmget.....) buffers in order to get
highest transfer speeds.
The buffers (>= 16 MByte) span over multiple of the 2 MByte pages.  
In cases where my read buffer is smaller then a certain size let's say
128 MByte I get  IBV_WC_LOC_PROT_ERR into the completion queue.
This happens for any RDMA transmission size.

The only visible difference in user space between the buffers that fail
and the ones that work seems to be their address returned from the shmat
command
The ones that work are out of the range 0x75E00000 (>= 128MByte) the
ones that fail are in the range 0xFC800000. (< 128 MByte)

I'm using OFED 1.1 here.

Any help/idea welcome.

Thomas


P.S.:	On Mellanox gen1 I already discovered a bug where only the first
page of the buffer was filled correctly. This was fixed in a patch for
IBGD-1.8.2
P.P.S:	To register my buffers I'm using:

accessFlags = (ibv_access_flags)(IBV_ACCESS_LOCAL_WRITE  |
                                 IBV_ACCESS_REMOTE_WRITE |
                                 IBV_ACCESS_REMOTE_READ  |
	                           IBV_ACCESS_REMOTE_ATOMIC|
	                           IBV_ACCESS_MW_BIND);

mr = ibv_reg_mr(_pd, (void*)(uintptr_t)bufferPtr, size, accessFlags);
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20061121/fa9ec413/attachment.html>