[ofa-general] OFEM installation problem

[Boyan Lazarov]

Hello,

Thanks for the answer! Yes, the problem was the memory locking for the users. 
The root does not have this problem.  

Best regards,
Boyan

On Sunday 28 June 2009 17:16:32 Tziporet Koren wrote:
> Boyan Lazarov wrote:
> > Hello,
> >
> > We have installed OFED  1.4.1 on Scientific Linux 53 (RH derivative).
> > When I run the osu_* tests  (OpenMPI) as root it seems that everything is
> > running fine and the mpi is using openib for communications.  However if
> > I run the test as an ordinary user I am getting the following:
> >
> > WARNING: There was an error initializing an OpenFabrics device.
> >
> > and openmpi is using tcp for communications - the speed is much lower.
> >
> > Anyone with similar problem and any suggestions?
>
> Can you send output of /var/log/messages?
> Also it could be that you have this problem:
> 2. Memory registration by the user is limited according to administrator
>    setting. See "Pinning (Locking) User Memory Pages" in OFED_tips.txt for
>    system configuration.
>
> ===========================================================================
>==== 3. Pinning (Locking) User Memory Pages
> ===========================================================================
>====
>
> Memory locking is managed by the kernel on a per user basis. Regular
> users (as
> opposed to root) have a limited number of pages which they may pin, where
> the limit is pre-set by the administrator. Registering memory for IB verbs
> requires pinning memory, thus an application cannot register more memory
> than
> it is allowed to pin.
>
> The user can change the system per-process memory lock limit by adding
> the following two lines to the file /etc/security/limits.conf:
>
>   *  soft  memlock  <number>
>   *  hard  memlock  <number>
>
>   where <number> denotes the number of KBytes that may be locked by a
>   user process.
>
> The above change to /etc/security/limits.conf will allow any user
> process in the
> system to lock up to <number> KBytes of memory.
>
> On some systems, it may be possible to use "unlimited" for the size to
> disable
> these limits entirely.
>
> Note: The file /etc/security/limits.conf contains further documentation.
>
> Tziporet