[openib-general] ipoib question when running on the same node as opensm

[Ira Weiny]

We just brought another cluster up and had an issue with our management node
(node running opensm) not coming up on ipoib.  Here is what happened and how I
got it working and I had some questions.

1) We had both opensm running and a switch based Voltaire SM running.  This
   caused problems.

2) We stopped the Voltaire SM and restarted all the nodes.  This got all of the
   nodes except the one with opensm running to work.

3) I had to unload all the modules, load only those needed by opensm, start
   opensm, and then bring up the ipoib interface.  At this point the node
   seemed to be in the multicast group and ipoib worked fine.

Does this seem like proper behavior?  I would think that on boot if ipoib does
not find a SM running it will delay setting up a connection until the SM comes
on-line?  (ie when the opensm init script gets run.)

It seems like the card saves some information (from the Voltaire SM) across a
soft reboot?  I know that it was not coming up in the multicast group with the
opensm.  Is this by design?

At this point ipoib seems to work fine after a reboot even though the interface
is brought up before opensm.  Do I need to ensure that opensm is up before all
ipoib requests in the future?

Thanks,
Ira Weiny
weiny2 at llnl.gov