[Users] increasing limit of Registerable memory with OFED-3.5-1

[Anton Starikov]

Hi,

there was a long discussion time ago on what to use. log_mtts_per_seg increases fragmentation.

But, overal problem is not to increase this limit, but because there is bug in 1.6.x and trunk.

To check available limit, openib btl checks /sys/module/mlx4_core/parameters/log_num_mtt , which is nonexistent in fresh drivers, as result of unavailable value, it assumes that log_num_mtt=20 (default value on older drivers).

Therefore, this check is irrelevant.

Anton.

PS: I have MT27500 (ConnectX-3).




On Jul 24, 2013, at 12:48 PM, "Roberto R. Expósito" <rreye at udc.es> wrote:

> Hi,
> 
> I think you can try with the log_mtts_per_seg parameter.
> 
> Regards,
> 
> Roberto R. Expósito
> 
> El 24/07/13 09:31, Anton Starikov escribió:
>> Hello,
>> 
>> 
>> I'm using OFED-3.5-1 with SL-6.4 (I had to do some minor patching to get it working on 2.6.32-358.14.1.el6.x86_64 kernel due to double export of __pskb_copy).
>> 
>> OpenMPI give known warning about limit of registrable memory (below). But in current modules there is no "log_num_mtt" parameter to tune.
>> 
>> Just in case, My hard and soft limits for maximal locked memory are unlimited.
>> 
>> What should be procedure with latest OFED then?
>> 
>> Thank you,
>> 
>> Anton Starikov
>> 
>> ------------------------------------------
>> 
>> WARNING: It appears that your OpenFabrics subsystem is configured to only
>> allow registering part of your physical memory.  This can cause MPI jobs to
>> run with erratic performance, hang, and/or crash.
>> 
>> This may be caused by your OpenFabrics vendor limiting the amount of
>> physical memory that can be registered.  You should investigate the
>> relevant Linux kernel module parameters that control how much physical
>> memory can be registered, and increase them to allow registering all
>> physical memory on your machine.
>> 
>> See this Open MPI FAQ item for more information on these Linux kernel module
>> parameters:
>> 
>>     http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
>> 
>>   Local host:              node1
>>   Registerable memory:     32768 MiB
>>   Total memory:            262098 MiB
>> 
>> Your MPI job will continue, but may be behave poorly and/or hang.
>> _______________________________________________
>> Users mailing list
>> Users at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users
>