[openib-general] ibv_reg_mr failure with pvfs on ehca?

[Troy Benjegerdes]

I am running PVFS2 on OpenIB, with IBM's ehca.

When we start writing/reading large files, either with the NetPIPE  
PVFS module we have or a modified GAMESS executable that uses  
libpvfs2 directly, the 'ibv_reg_mr' function fails, and we get an error.

This is also correlated with kernel log messages like this:

Oct 16 11:14:45 p5l8 kernel: PU0003 000e0091:ehca_hcall_7arg_7ret  
HCAD_ERROR  opco
de=160 ret=fffffffffffffff7 arg1=1000000003000004 arg2=5  
arg3=14f0ebc8 arg4=10000
arg5=e0000000000000 arg6=e3e9f200 arg7=0 out1=0 out2=0 out3=0 out4=0  
out5=0 out6=0
out7=0
Oct 16 11:14:45 p5l8 kernel: PU0003 00090454:ehca_reg_mr HCAD_ERROR   
hipz_alloc_mr
failed, h_ret=fffffffffffffff7 hca_hndl=1000000003000004
Oct 16 11:14:45 p5l8 kernel: PU0003 00090478:ehca_reg_mr <<<  
ret=ffffffea shca=c00
00000e796b000 e_mr=c0000000d22c7d80 iova_start=0000000014f0ebc8  
size=10000 acl=7 e
_pd=c0000000e3e9f200 pginfo=c0000001ad37fa70 num_pages=11 num_4k=11
Oct 16 11:14:45 p5l8 kernel: PU0003 00090176:ehca_reg_user_mr <<<  
rc=fffffffffffff
fea pd=c0000000e3e9f200 region=c0000000cb73a9d0 mr_access_flags=7  
udata=c0000001ad
37fba0


We are able to run on a 4x PCI-X Mellanox HCA, but obviously I'd like  
to be using the 12x ehca.