[ofa-general] OFEM installation problem

[Tziporet Koren]

Boyan Lazarov wrote:
> Hello,
>
> We have installed OFED  1.4.1 on Scientific Linux 53 (RH derivative). When I 
> run the osu_* tests  (OpenMPI) as root it seems that everything is running 
> fine and the mpi is using openib for communications.  However if I run the 
> test as an ordinary user I am getting the following:
>
> WARNING: There was an error initializing an OpenFabrics device.
>
> and openmpi is using tcp for communications - the speed is much lower. 
>
> Anyone with similar problem and any suggestions?
>
>   
Can you send output of /var/log/messages?
Also it could be that you have this problem:
2. Memory registration by the user is limited according to administrator
   setting. See "Pinning (Locking) User Memory Pages" in OFED_tips.txt for
   system configuration.

===============================================================================
3. Pinning (Locking) User Memory Pages
===============================================================================

Memory locking is managed by the kernel on a per user basis. Regular 
users (as
opposed to root) have a limited number of pages which they may pin, where
the limit is pre-set by the administrator. Registering memory for IB verbs
requires pinning memory, thus an application cannot register more memory 
than
it is allowed to pin.

The user can change the system per-process memory lock limit by adding
the following two lines to the file /etc/security/limits.conf:

  *  soft  memlock  <number>
  *  hard  memlock  <number>

  where <number> denotes the number of KBytes that may be locked by a
  user process.

The above change to /etc/security/limits.conf will allow any user 
process in the
system to lock up to <number> KBytes of memory.

On some systems, it may be possible to use "unlimited" for the size to 
disable
these limits entirely.

Note: The file /etc/security/limits.conf contains further documentation.

Tziporet