[openfabrics-ewg] rc3 installation issues.

Scott Weitzenkamp (sweitzen) sweitzen at cisco.com
Mon May 1 12:14:46 PDT 2006


We are using Intel MPI 2.0 (plan to upgrade to 2.0.1 soon) with "mpiexec
-genv I_MPI_DEBUG 3 -genv I_MPI_DEVICE rdma".  Are there any tunable
parameters we should be setting?
 
###TEST-D: MPI home dir is
/data/software/qa/MPI/intel_mpi/intelmpi-2.0-`uname -
m`.
 
###TEST-D: Intel-MPI uDAPL osu_latency test on hosts svbu-qa1850-1
svbu-qa1850-2
# OSU MPI Latency Test (Version 2.1)
# Size          Latency (us)
0               6.89
1               7.25
2               7.43
4               7.38
8               7.25
16              7.62
32              7.45
64              7.86
128             8.46
256             9.53
512             6.55
1024            7.29
2048            8.57
4096            11.46
8192            17.19
16384           29.26
32768           52.75
65536           82.67
131072          159.05
262144          282.31
524288          527.08
1048576         1019.57
2097152         1980.07
4194304         3921.89
###TEST-D: Intel-MPI uDAPL osu_bw test on hosts svbu-qa1850-1
svbu-qa1850-2
# OSU MPI Bandwidth Test (Version 2.1)
# Size          Bandwidth (MB/s)
1               0.426379
2               0.852988
4               1.711797
8               3.370452
16              6.725704
32              13.475464
64              25.936968
128             48.790506
256             93.297867
512             170.188285
1024            290.329228
2048            452.958568
4096            579.709399
8192            627.702319
16384           643.040963
32768           652.009193
65536           861.314972
131072          995.337153
262144          1112.127322
524288          1184.367387
1048576         1223.328478
2097152         1245.594018
4194304         1256.059934
###TEST-D: Intel-MPI uDAPL osu_bibw test on hosts svbu-qa1850-1
svbu-qa1850-2
# OSU MPI Bidirectional Bandwidth Test (Version 2.1)
# Size          Bi-Bandwidth (MB/s)
1               0.531761
2               1.063121
4               2.147913
8               4.280755
16              8.523452
32              16.981693
64              33.874987
128             64.956544
256             122.497886
512             226.783855
1024            367.798527
2048            531.763593
4096            613.404163
8192            638.258306
16384           645.862115
32768           955.762031
65536           1089.402315
131072          1134.346706
262144          1191.923405
524288          1227.324266
1048576         1245.633844
2097152         1254.414692
4194304         1261.847862

 
Scott Weitzenkamp
SQA and Release Manager
Server Virtualization Business Unit
Cisco Systems
 


________________________________

	From: Davis, Arlin R [mailto:arlin.r.davis at intel.com] 
	Sent: Monday, May 01, 2006 11:29 AM
	To: Scott Weitzenkamp (sweitzen); Woodruff, Robert J; Aviram
Gutman; Tziporet Koren; Amit Mehrotra (amehrotr);
openfabrics-ewg at openib.org
	Cc: Alexander Smirnov
	Subject: RE: [openfabrics-ewg] rc3 installation issues.
	
	

	Scott,

	 

	Thanks for testing Intel MPI on OFED 1.0 rc3. Yes, the socket CM
uDAPL provider will work fine without uCMA and IPoIB installed. Any idea
why your latency is so high? It should be around 3-4us.

	 

	-arlin

	 

	
________________________________


	From: Scott Weitzenkamp (sweitzen) [mailto:sweitzen at cisco.com] 
	Sent: Sunday, April 30, 2006 10:32 PM
	To: Woodruff, Robert J; Aviram Gutman; Tziporet Koren; Amit
Mehrotra (amehrotr); openfabrics-ewg at openib.org
	Cc: Alexander Smirnov; Davis, Arlin R
	Subject: RE: [openfabrics-ewg] rc3 installation issues.

	 

	Intel MPI 2.0 does work with OFED 1.0 rc3 on RHEL4 U3 despite
the lack of uCMA, here's example output on x86_64 w/PCI-E:

	 

	.../osu_latency.x
	CMA: unable to open /dev/infiniband/rdma_cm
	CMA: unable to open /dev/infiniband/rdma_cm
	CMA: unable to open /dev/infiniband/rdma_cm
	CMA: unable to open /dev/infiniband/rdma_cm
	I_MPI: [0] set_up_devices(): will use device: libmpi.rdma.so
	I_MPI: [0] set_up_devices(): will use DAPL provider: OpenIB-scm1
	CMA: unable to open /dev/infiniband/rdma_cm
	CMA: unable to open /dev/infiniband/rdma_cm
	CMA: unable to open /dev/infiniband/rdma_cm
	CMA: unable to open /dev/infiniband/rdma_cm
	I_MPI: [0] set_up_devices(): will use device: libmpi.rdma.so
	I_MPI: [0] set_up_devices(): will use DAPL provider: OpenIB-scm1
	# OSU MPI Latency Test (Version 2.1)
	# Size          Latency (us) 
	0               7.86
	1               7.53
	2               7.37
	4               7.43
	8               7.92
	16              7.29
	32              7.36
	64              7.50

	 

	Scott Weitzenkamp

	SQA and Release Manager

	Server Virtualization Business Unit

	Cisco Systems

	 

		 

		
________________________________


		From: openfabrics-ewg-bounces at openib.org
[mailto:openfabrics-ewg-bounces at openib.org] On Behalf Of Woodruff,
Robert J
		Sent: Tuesday, April 25, 2006 9:16 AM
		To: Aviram Gutman; Tziporet Koren; Amit Mehrotra
(amehrotr); openfabrics-ewg at openib.org
		Cc: Alexander Smirnov; Davis, Arlin R
		Subject: RE: [openfabrics-ewg] rc3 installation issues.

		Intel MPI needs a stable uDAPL and supporting usermode
verbs/uCMA and such.

		If you have that, then Intel MPI should work. If you
want to get a copy of Intel MPI

		to use in your testing of OFED, I can put you in contact
with our Intel MPI lead and he can

		provide you with the code and a license. 

		 

		
________________________________


		From: Aviram Gutman [mailto:aviram at mellanox.co.il] 
		Sent: Tuesday, April 25, 2006 9:09 AM
		To: Woodruff, Robert J; Tziporet Koren; Amit Mehrotra
(amehrotr); openfabrics-ewg at openib.org
		Cc: Alexander Smirnov; Davis, Arlin R
		Subject: RE: [openfabrics-ewg] rc3 installation issues.

		Please work with Vlad to make sure you have what you
need in rc4.

		 

		Regards,

		   Aviram

		 

		 

		
________________________________


		From: Woodruff, Robert J
[mailto:robert.j.woodruff at intel.com] 
		Sent: Tuesday, April 25, 2006 6:56 PM
		To: Aviram Gutman; Tziporet Koren; Amit Mehrotra
(amehrotr); openfabrics-ewg at openib.org
		Cc: Alexander Smirnov; Davis, Arlin R
		Subject: RE: [openfabrics-ewg] rc3 installation issues.

		Not yet. Last I heard, there was no uDAPL in OFED that
is neeed by Intel MPI. Maybe RC4. 

		 

		 

		 Tziporet wrote,

		> Regarding uDAPL and RHEL4 (up2 & 3): Since we have not
completed the uCMA backport for kernel 2.6.9 we eliminated uDAPL  

		 

		
________________________________


		From: Aviram Gutman [mailto:aviram at mellanox.co.il] 
		Sent: Tuesday, April 25, 2006 8:48 AM
		To: Woodruff, Robert J; Tziporet Koren; Amit Mehrotra
(amehrotr); openfabrics-ewg at openib.org
		Cc: Alexander Smirnov; Davis, Arlin R
		Subject: RE: [openfabrics-ewg] rc3 installation issues.

		Woody,

		 

		Are you testing Intel MPI on top of OFED?

		 

		Regards,

		   Aviram

		 

		 

		
________________________________


		From: openfabrics-ewg-bounces at openib.org
[mailto:openfabrics-ewg-bounces at openib.org] On Behalf Of Woodruff,
Robert J
		Sent: Monday, April 24, 2006 11:33 PM
		To: Tziporet Koren; Amit Mehrotra (amehrotr);
openfabrics-ewg at openib.org
		Cc: Alexander Smirnov; Davis, Arlin R
		Subject: RE: [openfabrics-ewg] rc3 installation issues.

		 

		The latest patches I have is for SVN 6509 on the trunk. 

		https://openib.org/svn/gen2/branches/backport-to-2.6.9/

	
infiniband-backport-svn6509-to-2.6.9-34.EL-kernel-fixups-00.diff
	
infiniband-backport-svn6509-to-2.6.9-34.EL-kernel-fixups-01.diff
	
infiniband-backport-svn6509-to-2.6.9-openib-drivers-02.diff
	
infiniband-backport-svn6509-to-2.6.9-openib-fixups-03.diff

		I don't split the patches by component, but rather have
one to strip out the old IB

		code in RedHat EL4.0, one to apply needed kernel changes
to the RedHat kernel,

		one for the openib drivers, and one for openib fixups
that are needed.

		 

		 

		 

		 

		 

		
________________________________


		From: Tziporet Koren [mailto:tziporet at mellanox.co.il] 
		Sent: Monday, April 24, 2006 11:43 AM
		To: Woodruff, Robert J; Amit Mehrotra (amehrotr);
openfabrics-ewg at openib.org
		Cc: Alexander Smirnov; Davis, Arlin R
		Subject: RE: [openfabrics-ewg] rc3 installation issues.

		We plan to do a backport of uCMA for RC4.

		Can you point me to your backport of uCMA for 2.6.9-34EL

		 

		Thanks,

		Tziporet

		 

		-----Original Message-----
		From: Bob Woodruff [mailto:robert.j.woodruff at intel.com] 
		Sent: Monday, April 24, 2006 6:56 PM
		To: Tziporet Koren; Amit Mehrotra (amehrotr);
openfabrics-ewg at openib.org
		Cc: Alexander Smirnov; Davis, Arlin R
		Subject: RE: [openfabrics-ewg] rc3 installation issues.

		 

		 Tziporet wrote,

		> Regarding uDAPL and RHEL4 (up2 & 3): Since we have not
completed the uCMA backport for kernel 2.6.9 we eliminated uDAPL  

		>  installation in case of this kernel. We also
indicated this in the package limitation mail. 

		 

		Note that there are two uDAPL providers for OpenIB. One
uses standard sockets for connections (socket CM)

		and the other uses CMA. I would suggest that you should
include the socket CM uDAPL provider even

		if you do not have a backport for uCMA.

		BTW. - My backport patches contain backport for uCMA for
2.6.9-34-EL if you need an example.

		 

		woody

		 

		 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ewg/attachments/20060501/c39a777d/attachment.html>


More information about the ewg mailing list