[Users] increasing limit of Registerable memory with OFED-3.5-1

[Anton Starikov]

Hello, 


I'm using OFED-3.5-1 with SL-6.4 (I had to do some minor patching to get it working on 2.6.32-358.14.1.el6.x86_64 kernel due to double export of __pskb_copy).

OpenMPI give known warning about limit of registrable memory (below). But in current modules there is no "log_num_mtt" parameter to tune.

Just in case, My hard and soft limits for maximal locked memory are unlimited.

What should be procedure with latest OFED then?

Thank you,

Anton Starikov

------------------------------------------

WARNING: It appears that your OpenFabrics subsystem is configured to only
allow registering part of your physical memory.  This can cause MPI jobs to
run with erratic performance, hang, and/or crash.

This may be caused by your OpenFabrics vendor limiting the amount of
physical memory that can be registered.  You should investigate the
relevant Linux kernel module parameters that control how much physical
memory can be registered, and increase them to allow registering all
physical memory on your machine.

See this Open MPI FAQ item for more information on these Linux kernel module
parameters:

    http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages

  Local host:              node1
  Registerable memory:     32768 MiB
  Total memory:            262098 MiB

Your MPI job will continue, but may be behave poorly and/or hang.