[ofa-general] Performance of UDAPL RDMA vs IB verbs

[Chuck Hartley]

Scott-

Here is what we have:

Cisco BladeCenterH> show version

================================================================================
                           System Version Information
================================================================================
           system-version : Cisco-BCH TopspinOS 2.6.0 releng #195 06/01/2006
01:32:43 (UUID:00000000-0000-0000-0005-AD0000094076)


On Feb 18, 2008 12:12 PM, Scott Weitzenkamp (sweitzen) <sweitzen at cisco.com>
wrote:

>  Chuck, the SM in the Topspin/Cisco switch can be turned off from CLI, Web
> UI, or Element Manager UI.
>
> There was a SM bug last year where DDR throughput was not as high as it
> should be sometimes, what does "show version" display from the Topspin/Cisco
> switch CLI?
>
>
> Scott Weitzenkamp
> SQA and Release Manager
> Server Virtualization Business Unit
> Cisco Systems
>
>
>  ------------------------------
> *From:* general-bounces at lists.openfabrics.org [mailto:
> general-bounces at lists.openfabrics.org] *On Behalf Of *Chuck Hartley
> *Sent:* Monday, February 18, 2008 7:19 AM
> *To:* Hal Rosenstock
> *Cc:* OpenFabrics General
> *Subject:* Re: [ofa-general] Performance of UDAPL RDMA vs IB verbs
>
> Thanks for the tips on the commands - that was my next question.
>
> > Is the rest of the subnet pure DDR and a mix of SDR/DDR ?
>
> The rest of the network is a mixture of SDR/DDR.  Here is what we have:
>
> ]# ibnodes -V
> Ca      : 0x0005ad00001dc9e4 ports 2 " HCA-1"
> Ca      : 0x0005ad00001dc970 ports 2 " HCA-1"
> Ca      : 0x0020c28001067759 ports 1 "IB900 TMS Infiniband interface"
> Ca      : 0x0020c28002067759 ports 1 "IB900 TMS Infiniband interface"
> Ca      : 0x0020c28003067759 ports 1 "IB900 TMS Infiniband interface"
> Ca      : 0x0020c28004067759 ports 1 "IB900 TMS Infiniband interface"
> Ca      : 0x0030487a06600000 ports 1 "Linux62 HCA-1"
> Ca      : 0x0030487a226a0000 ports 1 "linux64 HCA-1"
> Ca      : 0x0030487a071e0000 ports 1 "Linux61 HCA-1"
> Ca      : 0x0002c9020023bb14 ports 1 "Linux60 HCA-1"
> Ca      : 0x0030487a2a480000 ports 1 "Linux63 HCA-1"
> Switch  : 0x000b8cffff00441c ports 24 "MT47396 Infiniscale-III Mellanox
> Technologies" base port 0 lid 3 lmc 0
> Switch  : 0x0005ad0000094076 ports 24 "Topspin Switch" enhanced port 0 lid
> 15 lmc 0
>
> The first two nodes in the list are the SDR blades off of the Topspin
> switch.  The Linux boxes are all Mellanox DDR HCAs and the IB900 interfaces
> are SDR. The Linux and IB900 ports are all connected to the Mallanox
> switch.  The Topspin switch is connected to the Mellanox switch.
>
>
> > Does it work "right" with OpenSM off/using the vendor SM ?
>
> No, it does not work with only the SM in the Topspin switch running.  That
> was the initial configuration we had.  Then I started OpenSM on one of the
> Linux boxes and that did not work any better.  I also tried disconnection
> the TMS IB900 so there was only DDR devices on the subnet and that did not
> make a difference either.  It seems like the SM in the Topspin switch only
> understands SDR maybe.  There does not appear to be an option for turning
> off the SM in the Topspin switch, however you can change its priority
> relative to other SM on the subnet.  That is what I did and now the OpenSM
> is the master.
>
> I didn't see anything in the opensm.conf file that indicates that OpenSM
> has a concept of priority.  Is there some way to force it to always be the
> master?  Is there some advantage or disadvantage to running multiple copies
> of OpenSM on the subnet?   If you have multiple switches connected as we do,
> should some of the default settings on opensm.conf be changed? In
> particular, should REASSIGN_LIDS be set to "yes"?
>
> Chuck
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20080218/956a7fb4/attachment.html>