[openib-general] Running MVAPICH2 with SLURM Process Manager

[Sayantan Sur]

* On May,3 Roland Dreier<rdreier at cisco.com> wrote :
>     Ira> I have been meaning to ping the list to see if anyone else
>     Ira> has this issue and if there are any ideas on how to keep a
>     Ira> limit on max locked memory while still allowing MVAPICH to
>     Ira> run.
> 
> Not really -- registering memory for use with RDMA requires that it be
> locked, and MVAPICH wants to register a lot of memory.  I guess
> setting a large limit would work too, but I don't know exactly how
> large the limit would have to be -- and it would probably depend on
> what MPI codes you're running.

The amount of registered memory in MVAPICH is very much application
dependent and easily tunable at runtime. There are three main parameters
which you can pass to MVAPICH to control the amount of registered
memory.

VIADEV_DREG_CACHE_LIMIT: Controls the limit of pages reserved for large
messages.
VIADEV_VBUF_POOL_SIZE: Controls the number of buffers allocated for
small messages.
VIADEV_NUM_RDMA_BUFFER: Controls the number of RDMA buffers for scalable
RDMA channel for small messages

All these parameters are explained in detail in our user guide:

http://nowlab.cse.ohio-state.edu/projects/mpi-iba/mvapich_user_guide.html#x1-540009

Based on the tuning of the MVAPICH library and the application codes you
are running, you may decide on a lower ulimit than just "unlimited".

Thanks,
Sayantan.


-- 
http://www.cse.ohio-state.edu/~surs