[ofa-general] Performance of UDAPL RDMA vs IB verbs

[Scott Weitzenkamp (sweitzen)]

Chuck, the SM in the Topspin/Cisco switch can be turned off from CLI,
Web UI, or Element Manager UI. 
 
There was a SM bug last year where DDR throughput was not as high as it
should be sometimes, what does "show version" display from the
Topspin/Cisco switch CLI?
 
Scott Weitzenkamp
SQA and Release Manager
Server Virtualization Business Unit
Cisco Systems


 


________________________________

	From: general-bounces at lists.openfabrics.org
[mailto:general-bounces at lists.openfabrics.org] On Behalf Of Chuck
Hartley
	Sent: Monday, February 18, 2008 7:19 AM
	To: Hal Rosenstock
	Cc: OpenFabrics General
	Subject: Re: [ofa-general] Performance of UDAPL RDMA vs IB verbs
	
	
	Thanks for the tips on the commands - that was my next question.
	
	> Is the rest of the subnet pure DDR and a mix of SDR/DDR ?
	
	The rest of the network is a mixture of SDR/DDR.  Here is what
we have:
	
	]# ibnodes -V
	Ca      : 0x0005ad00001dc9e4 ports 2 " HCA-1"
	Ca      : 0x0005ad00001dc970 ports 2 " HCA-1"
	Ca      : 0x0020c28001067759 ports 1 "IB900 TMS Infiniband
interface"
	Ca      : 0x0020c28002067759 ports 1 "IB900 TMS Infiniband
interface"
	Ca      : 0x0020c28003067759 ports 1 "IB900 TMS Infiniband
interface"
	Ca      : 0x0020c28004067759 ports 1 "IB900 TMS Infiniband
interface"
	Ca      : 0x0030487a06600000 ports 1 "Linux62 HCA-1"
	Ca      : 0x0030487a226a0000 ports 1 "linux64 HCA-1"
	Ca      : 0x0030487a071e0000 ports 1 "Linux61 HCA-1"
	Ca      : 0x0002c9020023bb14 ports 1 "Linux60 HCA-1"
	Ca      : 0x0030487a2a480000 ports 1 "Linux63 HCA-1"
	Switch  : 0x000b8cffff00441c ports 24 "MT47396 Infiniscale-III
Mellanox Technologies" base port 0 lid 3 lmc 0
	Switch  : 0x0005ad0000094076 ports 24 "Topspin Switch" enhanced
port 0 lid 15 lmc 0
	
	The first two nodes in the list are the SDR blades off of the
Topspin switch.  The Linux boxes are all Mellanox DDR HCAs and the IB900
interfaces are SDR. The Linux and IB900 ports are all connected to the
Mallanox switch.  The Topspin switch is connected to the Mellanox
switch.
	
	
	> Does it work "right" with OpenSM off/using the vendor SM ?
	
	No, it does not work with only the SM in the Topspin switch
running.  That was the initial configuration we had.  Then I started
OpenSM on one of the Linux boxes and that did not work any better.  I
also tried disconnection the TMS IB900 so there was only DDR devices on
the subnet and that did not make a difference either.  It seems like the
SM in the Topspin switch only understands SDR maybe.  There does not
appear to be an option for turning off the SM in the Topspin switch,
however you can change its priority  relative to other SM on the subnet.
That is what I did and now the OpenSM  is the master.
	
	I didn't see anything in the opensm.conf file that indicates
that OpenSM has a concept of priority.  Is there some way to force it to
always be the master?  Is there some advantage or disadvantage to
running multiple copies of OpenSM on the subnet?   If you have multiple
switches connected as we do, should some of the default settings on
opensm.conf be changed? In particular, should REASSIGN_LIDS be set to
"yes"?
	
	Chuck
	
	

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20080218/b64b1a0c/attachment.html>