[openib-general] ibv_reg_mr failure with pvfs on ehca?

[Hoang-Nam Nguyen]

Hello Troy and Kyle!
> > Kyle wrote:
> > Our app writes out a file once, then reads it in many times through
> > the pvfs2 system.  In the pvfs2 layers, there is memory caching
> > done at the network level, so memory is registered by the app, and
> > attempts are made to re-register and/or re-use these memory regions
> > to save on memory reg overhead.  The problem occurs only while
> > writing files, so while memory is being initially registered with
> > the nic/app and cached?  Also, our tests show that the app runs
> > normally to completion on identical machines using mellanox hca's
> > instead of the eHCA.  The file sizes are generally >16GByte,
> > however our failures usually appear by the time ~220-250MBytes have
> > been written(possibly also all registered)?
We have tested memory registration with 64GB. So I don't think ~16GB
is an issue. However we do have a restriction of mappings in that the
total number of mappings is twice of the total number of pages assigned
to the partition. The term mappings means the number of pages in the
calls to ib_reg_phys_mr() or ib_reg_user_mr()/ibv_reg_mr(). ehca driver
does register the whole space at module load time so that for user
space applications you have a limit of mappings equal the total number
of physical pages. Note that kernel modules sitting on top of ehca
eg. ib_ipoib, ib_mad don't suffer under this limit since they share
the whole space registered by ehca as they call ib_get_dma_mr().
> > I'm not sure the standard OpenIB NetPIPE runs can reproduce this
> > type of workload.  However, we have developed a working PVFS2-
> > NetPIPE module which can reproduce this problem on occassion, if
> > there is interest in further testing this on your end, I can make
> > it available.
Yes. Please send it to me. I'd like to test it. Is it a user space appl.?
I want to see if we could reach the limit of mappings mentioned above.
> > Our ehca's have the following revision info:
> >        vendor_id:                      0x5076
> >        vendor_part_id:                 0
> >        hw_ver:                         0x1000003
> > Kernel version is debian 2.6.17
ok. For completeness please give me the driver version using
modinfo and also the firmware code level via HMC - click on "Licensed
Internal Code Maintenance" (left pane), "Change Licensed Internal
Code" (right pane), select your frame and then "View System Info",
"Display Current Values".
You can also turn on the debug traces of ehca to track all reg_mr()
calls in order to determine if you reach the limit of mappings
mentioned above. Or just send me the whole dmesg resp.
/var/log/messages.
> Troy wrote:
> What are the limits on the ehca memory registrations?
> Is there a limit to the number of regions that can be registered?
See above
> Is
> there any way (with kernel hacks) that we can register the entire
> address space of the application? We would like to be able to do RDMA
> sends and receives from anywhere in the application address space
> eventually, and only register it once.
ib_ipoib or ib_mad actually uses ib_get_dma_mr() and obtain the whole
space. For user space there is no corresponding api yet.
> What is the point of RDMA for memory-intensive applications if you
> have to copy the data to a registered buffer before sending it anyway?
Not sure if I understand completely... However RDMA mr accesses should
not be an issue.
Thanks!
Nam