[openib-general] ibv_reg_mr failure with pvfs on ehca?

[Troy Benjegerdes]

>>> I'm not sure the standard OpenIB NetPIPE runs can reproduce this
>>> type of workload.  However, we have developed a working PVFS2-
>>> NetPIPE module which can reproduce this problem on occassion, if
>>> there is interest in further testing this on your end, I can make
>>> it available.
> Yes. Please send it to me. I'd like to test it. Is it a user space  
> appl.?
> I want to see if we could reach the limit of mappings mentioned above.

The netpipe code is available with mercurial by:

hg clone http://source.scl.ameslab.gov/hg/netpipe3-pvfs-dev

Once you have pvfs2-1.5.1 installed, you should be able to do 'make  
pvfs' in the netpipe3-pvfs-dev directory and build NPpvfs.

The command line arguments I used to reproduce this were:

./NPpvfs -d $PVFS_FILE_PATH -l 32768 -u 268435456 -n 100 -o  
$NETPIPE_OUTPUT_FILE


This is the dmesg log:

PU0001 000e0091:ehca_hcall_7arg_7ret HCAD_ERROR  opcode=160  
ret=fffffffffffffff7 arg1=1000000003000004 arg2=5 arg3=4000f830000  
arg4=10000 arg5=e0000000000000 arg6=eb6b6920 arg7=0 out1=0 out2=0  
out3=0 out4=0 out5=0 out6=0 out7=0
PU0001 00090454:ehca_reg_mr HCAD_ERROR  hipz_alloc_mr failed,  
h_ret=fffffffffffffff7 hca_hndl=1000000003000004
PU0001 00090478:ehca_reg_mr <<< ret=ffffffea shca=c0000000e796b000  
e_mr=c0000000ce865e80 iova_start=000004000f830000 size=10000 acl=7  
e_pd=c0000000eb6b6920 pginfo=c0000000dfcb3a70 num_pages=10 num_4k=10
PU0001 00090176:ehca_reg_user_mr <<< rc=ffffffffffffffea  
pd=c0000000eb6b6920 region=c0000000ce861dd0 mr_access_flags=7  
udata=c0000000dfcb3ba0