[openib-general] Running MVAPICH2 with SLURM Process Manager

[Ira Weiny]

On Thu, 25 May 2006 08:55:00 -0700
Don.Dhondt at Bull.com wrote:

> 
> We made a couple attempts at rebuilding mvapich2 and our symptoms
> changed. Maybe
> for the better, but still not good results. In our last attempt we 
> disabled the compile option
> "USE_MPD_RING"  (HAVE_MPD_RING=""). It seemed to get further but then 
> failed with a 
> "cannot create cq" error message. We are obviously failing now in the 
> infiniband code.
> The perplexing thing is that the applications work when run with
> mpiexec (outside of slurm)
> and have the MPD deamons running.
> 
> The latest suggestion from LLNL is to make sure we have unlimited max 
> locked
> memory for our MPI tasks with:
> 
>  srun sh -c 'ulimit -l'
> 

Here at LLNL we are only running MVAPICH but yes we have had to run our slurm
with "ulimit -l unlimited".

I have been meaning to ping the list to see if anyone else has this issue and
if there are any ideas on how to keep a limit on max locked memory while still
allowing MVAPICH to run.

Ira Weiny
weiny2 at llnl.gov