[ofa-general] Performance of UDAPL RDMA vs IB verbs

[Chuck Hartley]

Thanks for the tips on the commands - that was my next question.

> Is the rest of the subnet pure DDR and a mix of SDR/DDR ?

The rest of the network is a mixture of SDR/DDR.  Here is what we have:

]# ibnodes -V
Ca      : 0x0005ad00001dc9e4 ports 2 " HCA-1"
Ca      : 0x0005ad00001dc970 ports 2 " HCA-1"
Ca      : 0x0020c28001067759 ports 1 "IB900 TMS Infiniband interface"
Ca      : 0x0020c28002067759 ports 1 "IB900 TMS Infiniband interface"
Ca      : 0x0020c28003067759 ports 1 "IB900 TMS Infiniband interface"
Ca      : 0x0020c28004067759 ports 1 "IB900 TMS Infiniband interface"
Ca      : 0x0030487a06600000 ports 1 "Linux62 HCA-1"
Ca      : 0x0030487a226a0000 ports 1 "linux64 HCA-1"
Ca      : 0x0030487a071e0000 ports 1 "Linux61 HCA-1"
Ca      : 0x0002c9020023bb14 ports 1 "Linux60 HCA-1"
Ca      : 0x0030487a2a480000 ports 1 "Linux63 HCA-1"
Switch  : 0x000b8cffff00441c ports 24 "MT47396 Infiniscale-III Mellanox
Technologies" base port 0 lid 3 lmc 0
Switch  : 0x0005ad0000094076 ports 24 "Topspin Switch" enhanced port 0 lid
15 lmc 0

The first two nodes in the list are the SDR blades off of the Topspin
switch.  The Linux boxes are all Mellanox DDR HCAs and the IB900 interfaces
are SDR. The Linux and IB900 ports are all connected to the Mallanox
switch.  The Topspin switch is connected to the Mellanox switch.


> Does it work "right" with OpenSM off/using the vendor SM ?

No, it does not work with only the SM in the Topspin switch running.  That
was the initial configuration we had.  Then I started OpenSM on one of the
Linux boxes and that did not work any better.  I also tried disconnection
the TMS IB900 so there was only DDR devices on the subnet and that did not
make a difference either.  It seems like the SM in the Topspin switch only
understands SDR maybe.  There does not appear to be an option for turning
off the SM in the Topspin switch, however you can change its priority
relative to other SM on the subnet.  That is what I did and now the OpenSM
is the master.

I didn't see anything in the opensm.conf file that indicates that OpenSM has
a concept of priority.  Is there some way to force it to always be the
master?  Is there some advantage or disadvantage to running multiple copies
of OpenSM on the subnet?   If you have multiple switches connected as we do,
should some of the default settings on opensm.conf be changed? In
particular, should REASSIGN_LIDS be set to "yes"?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20080218/249bef29/attachment.html>