Dear Jonathan Perkins,<br><br> you put me on the right track ! It was just a problem of memory locked, DAMN IT !<br><br> My /etc/security/limits.conf was set correctly with these lines :<br>* hard memlock unlimited <br>
* soft memlock unlimited<br><br>BUT when I was running "ulimit -l" as a user, I was getting "64" instead of "unlimited".<br><br>In order to have "unlimited" for all my shells, I had to put in the file /etc/ssh/sshd_config the line:<br>
<br>UsePAM yes<br><br>(and restart my sshd daemon : <br>systemctl restart sshd.service)<br><br> And now my MPI stack over infiniband is working as expected :D:D<br><br> Many many thanks again !<br><br> Jean-Charles<br><br>
<div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Dark Charlot</b> <span dir="ltr"><<a href="mailto:jcldc13@gmail.com">jcldc13@gmail.com</a>></span><br>Date: 2012/6/15<br>
Subject: Re: [ewg] OFED drivers or linux stock drivers ?<br>To: Jonathan Perkins <<a href="mailto:perkinjo@cse.ohio-state.edu">perkinjo@cse.ohio-state.edu</a>><br><br><br> HI,<br><br> after recompiling MVAPICH2 with your configure options, I got this:<br>
<br> mpirun_rsh -np 2 amos kerkira ./osu_bw<div class="im"><br><br><br>[cli_0]: aborting job:<br>Fatal error in MPI_Init:<br></div>Other MPI error, error stack:<br>
MPIR_Init_thread(408).......: <br>MPID_Init(296)..............: channel initialization failed<br>MPIDI_CH3_Init(283).........: <br>MPIDI_CH3I_RDMA_init(172)...: <br>rdma_setup_startup_ring(431): cannot create cq<br><br>[amos:mpispawn_0][readline] Unexpected End-Of-File on file descriptor 6. MPI process died?<br>
[amos:mpispawn_0][mtpmi_processops] Error while reading PMI socket. MPI process died?<br>[amos:mpispawn_0][child_handler] MPI process (rank: 0, pid: 11879) exited with status 1<div class="im"><br>[cli_1]: aborting job:<br>
Fatal error in MPI_Init:<br></div>
Other MPI error, error stack:<br>MPIR_Init_thread(408).......: <br>MPID_Init(296)..............: channel initialization failed<br>MPIDI_CH3_Init(283).........: <br>MPIDI_CH3I_RDMA_init(172)...: <br>rdma_setup_startup_ring(431): cannot create cq<br>
<br>[kerkira:mpispawn_1][readline] Unexpected End-Of-File on file descriptor 5. MPI process died?<br>[kerkira:mpispawn_1][mtpmi_processops] Error while reading PMI socket. MPI process died?<br>[kerkira:mpispawn_1][child_handler] MPI process (rank: 1, pid: 565) exited with status 1<br>
[kerkira:mpispawn_1][report_error] connect() failed: Connection refused (111)<br>[kerkira:mpispawn_1][report_error] connect() failed: Connection refused (111)<br><br> Thanks, JC<div class="HOEnZb"><div class="h5"><br><br>
<br><div class="gmail_quote">2012/6/15 Jonathan Perkins <span dir="ltr"><<a href="mailto:perkinjo@cse.ohio-state.edu" target="_blank">perkinjo@cse.ohio-state.edu</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">This could be something as simple as a locked limit issue. Can you<br>
rebuild mvapich2 by passing `--disable-fast --enable-g=dbg' to<br>
configure? You should get more useful output with these options.<br>
<br>
I'm cc'ing mvapich-discuss as well as this may be specific to MVAPICH2.<br>
<div><div><br>
On Thu, Jun 14, 2012 at 4:14 PM, Dark Charlot <<a href="mailto:jcldc13@gmail.com" target="_blank">jcldc13@gmail.com</a>> wrote:<br>
> Dear experts,<br>
><br>
> I am running mageia2 linux distribution which comes with kernel 3.3.6.<br>
><br>
> I downloaded ofed 1.5.4.1 drivers and compiled and installed (** with a lot<br>
> of pains and spec files modifications **) some of the RPM :<br>
><br>
> infiniband-diags-1.5.13-1.x86_64.rpm<br>
> infiniband-diags-debug-1.5.13-1.x86_64.rpm<br>
> libibmad-1.3.8-1.x86_64.rpm<br>
> libibmad-debug-1.3.8-1.x86_64.rpm<br>
> libibmad-devel-1.3.8-1.x86_64.rpm<br>
> libibmad-static-1.3.8-1.x86_64.rpm<br>
> libibumad-1.3.7-1.x86_64.rpm<br>
> libibumad-debug-1.3.7-1.x86_64.rpm<br>
> libibumad-devel-1.3.7-1.x86_64.rpm<br>
> libibumad-static-1.3.7-1.x86_64.rpm<br>
> libibverbs-1.1.4-1.24.gb89d4d7.x86_64.rpm<br>
> libibverbs-debug-1.1.4-1.24.gb89d4d7.x86_64.rpm<br>
> libibverbs-devel-1.1.4-1.24.gb89d4d7.x86_64.rpm<br>
> libibverbs-devel-static-1.1.4-1.24.gb89d4d7.x86_64.rpm<br>
> libibverbs-utils-1.1.4-1.24.gb89d4d7.x86_64.rpm<br>
> libmlx4-1.0.1-1.20.g6771d22.x86_64.rpm<br>
> libmlx4-debug-1.0.1-1.20.g6771d22.x86_64.rpm<br>
> libmlx4-devel-1.0.1-1.20.g6771d22.x86_64.rpm<br>
> mstflint-1.4-1.18.g1adcfbf.x86_64.rpm<br>
> mstflint-debug-1.4-1.18.g1adcfbf.x86_64.rpm<br>
> opensm-3.3.13-1.x86_64.rpm<br>
> opensm-debug-3.3.13-1.x86_64.rpm<br>
> opensm-devel-3.3.13-1.x86_64.rpm<br>
> opensm-libs-3.3.13-1.x86_64.rpm<br>
> opensm-static-3.3.13-1.x86_64.rpm<br>
><br>
> But I was **not** able to compile ofa kernel itself.<br>
><br>
> Then I tried to use, instead, all the corresponding modules which come with<br>
> my stock linux kernel distribution (3.3.6)<br>
><br>
> After initializing correctly (I guess) all the necessary mellanox stuffs<br>
> (openibd, opensm etc...) I can see my Mellanox cards with the command<br>
> ibv_devinfo.<br>
><br>
> I get the following output for all the computers which have a mellanox card<br>
><br>
> 1) ibv_devinfo<br>
><br>
> kerkira:% ibv_devinfo<br>
><br>
> hca_id: mlx4_0<br>
> transport: InfiniBand (0)<br>
> fw_ver: 2.7.000<br>
> node_guid: 0002:c903:0009:d1b2<br>
> sys_image_guid: 0002:c903:0009:d1b5<br>
> vendor_id: 0x02c9<br>
> vendor_part_id: 26428<br>
> hw_ver: 0xA0<br>
> board_id: MT_0C40110009<br>
> phys_port_cnt: 1<br>
> port: 1<br>
> state: PORT_ACTIVE (4)<br>
> max_mtu: 2048 (4)<br>
> active_mtu: 2048 (4)<br>
> sm_lid: 8<br>
> port_lid: 8<br>
> port_lmc: 0x00<br>
> link_layer: IB<br>
><br>
><br>
> 2) ibstatus<br>
><br>
> kerkira:% /usr/sbin/ibstatus<br>
><br>
> Infiniband device 'mlx4_0' port 1 status:<br>
> default gid: fe80:0000:0000:0000:0002:c903:0009:d1b3<br>
> base lid: 0x8<br>
> sm lid: 0x8<br>
> state: 4: ACTIVE<br>
> phys state: 5: LinkUp<br>
> rate: 40 Gb/sec (4X QDR)<br>
> link_layer: InfiniBand<br>
><br>
><br>
> QUESTION:<br>
><br>
> ==> According to these outputs, could we say that my computers use correctly<br>
> the mlx4 drivers which comes with my kernel 3.3.6 ?<br>
><br>
><br>
> Probably not because I cannot communicate between two machines using<br>
> mpi.....<br>
><br>
> Here is the detail:<br>
> I compiled and install MVAPICH2 but I couldn't run "osu_bw" program between<br>
> two machines, I get :<br>
><br>
> kerkira% mpirun_rsh -np 2 kerkira amos ./osu_bw<br>
><br>
> [cli_0]: aborting job:<br>
> Fatal error in MPI_Init:<br>
> Other MPI error<br>
><br>
> [kerkira:mpispawn_0][readline] Unexpected End-Of-File on file descriptor 6.<br>
> MPI process died?<br>
> [kerkira:mpispawn_0][mtpmi_processops] Error while reading PMI socket. MPI<br>
> process died?<br>
> [kerkira:mpispawn_0][child_handler] MPI process (rank: 0, pid: 5396) exited<br>
> with status 1<br>
> [cli_1]: aborting job:<br>
> Fatal error in MPI_Init:<br>
> Other MPI error<br>
><br>
> [amos:mpispawn_1][readline] Unexpected End-Of-File on file descriptor 5. MPI<br>
> process died?<br>
> [amos:mpispawn_1][mtpmi_processops] Error while reading PMI socket. MPI<br>
> process died?<br>
> [amos:mpispawn_1][child_handler] MPI process (rank: 1, pid: 6733) exited<br>
> with status 1<br>
> [amos:mpispawn_1][report_error] connect() failed: Connection refused (111)<br>
><br>
><br>
> Now f I run on the **same** machine, I get the expected results:<br>
><br>
> kerkira% mpirun_rsh -np 2 kerkira kerkira ./osu_bw<br>
> # OSU MPI Bandwidth Test v3.6<br>
> # Size Bandwidth (MB/s)<br>
> 1 5.47<br>
> 2 11.34<br>
> 4 22.84<br>
> 8 45.89<br>
> 16 91.52<br>
> 32 180.27<br>
> 64 350.68<br>
> 128 661.78<br>
> 256 1274.94<br>
> 512 2283.42<br>
> 1024 3936.39<br>
> 2048 6362.91<br>
> 4096 9159.54<br>
> 8192 10737.42<br>
> 16384 9246.39<br>
> 32768 8869.26<br>
> 65536 8707.28<br>
> 131072 8942.07<br>
> 262144 9009.39<br>
> 524288 9060.31<br>
> 1048576 9080.17<br>
> 2097152 5702.06<br>
><br>
> (note: ssh between the machines kerkira and amos works correctly without<br>
> password)<br>
><br>
> QUESTION:<br>
><br>
> ==> Why MPI programs does not work between two machines ?<br>
> ==> Is it because I use the mlx4/umad/etc modules from my distribution<br>
> kernel and not OFED kernel-ib ?<br>
><br>
> Thanks in advance for your help .<br>
><br>
> Jean-Charles Lambert.<br>
><br>
><br>
><br>
</div></div>> _______________________________________________<br>
> ewg mailing list<br>
> <a href="mailto:ewg@lists.openfabrics.org" target="_blank">ewg@lists.openfabrics.org</a><br>
> <a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg</a><br>
<span><font color="#888888"><br>
<br>
<br>
--<br>
Jonathan Perkins<br>
<a href="http://www.cse.ohio-state.edu/%7Eperkinjo" target="_blank">http://www.cse.ohio-state.edu/~perkinjo</a><br>
</font></span></blockquote></div><br>
</div></div></div><br>