<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: arial,helvetica,sans-serif; font-size: 10pt; color: #000000'><hr id="zwchr"><div style="color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"John Valdes" <valdes@anl.gov><br><b>To: </b>"Andrei Mikhailovsky" <andrei@arhont.com><br><b>Cc: </b>users@lists.openfabrics.org<br><b>Sent: </b>Friday, 24 May, 2013 8:46:18 PM<br><b>Subject: </b>Re: [Users] HP BLc QLogic 4X QDR IB Switch oddness<br><br>Andrei,<br><br>On Fri, May 24, 2013 at 01:51:00PM +0100, Andrei Mikhailovsky wrote:<br>> <br>> I was wondering if anyone on the mailing list has experienced an odd behaviour with HP BLc QLogic 4X QDR IB Switch (505958-B21)? [...]<br>> <br>> My problem is that every time I reboot the blade servers which are connected to the above blade switch the switch ports take ages to Activate. [...]<br>> <br>> However, the server which are connected to the same switch with a cable (not interconnected blade servers) do not have this issue and get port state Active in seconds every time. <br><br>This last fact, that servers directly attached to the switch w/ a<br>cable negotiate link immediately, I think clearly points to the blade<br>servers and/or blade chassis as the source of the problem and not the<br>switch. <br><br><br>AM: it was my initial thoughts as well, but after trying two different switches and two different blade enclosures with the same results I thought otherwise and unsure.<br><br><br><br>What's the physical topology of the IB network between the blade<br>servers and the switch? I'm not familiar w/ the HP BLc, but I'm<br>guessing each blade server has an internal IB Mezzanine HCA which<br>connects to a chassis backplane, correct? How does the IB from the<br>backplane connect to the switch? Is there specialized connector<br>between them, or does it use a standard IB cable (possibly more than<br>one)? Does the chassis backplane have an IB switch chip?<br><br>AM: I am not really sure. The servers do have the IB mezzanine card and from what i've read it is a pci-e card. I am unsure how the blade servers are connected to the switch. I guess it's an internal HP/QLogic interconnect.<br><br><br><br>Some random thoughts/questions:<br><br>1) Where is the subnet manager running? What routing protocol is it<br> using (eg, minhop, updown, etc)?<br><br><br><br>AM: subnet manager is currently running from one of the servers connected externally to the switch via the IB cable. I did try to run subnet manager from the actual blade servers and the behaviour did not change.<br><br><br><br><br>2) When rebooting the blade servers, are you rebooting all the ones in<br> the same chassis at the same time? If you stagger the reboot of<br> each server, or if you just reboot one server, does it still take a<br> long time for it to negotiate link?<br><br><br><br>AM: tried rebooting one by one and all at the same time. No difference in results between the tries.<br><br><br><br><br>3) The problem could be a firmware issue; since the directly attahed<br> servers and the blade servers must have different HCAs (the former<br> standard PCI HCAs and the latter mezzanine HCAs I'm assuming), that<br> suggests they'll have different firmware. Unforutnately, you say<br> you're running the latest firmware already.<br><br><br>AM: from what I can tell, the IB Mezz hca doesn't have a firmware upgrade option. the firware shown is 0.0.0 and when I run the ofed driver install it tells me that my hca doesn't need a firmware upgrade. The switch is running the latest firmware from QLogic / Intel website. I think it's 7.0.0.35.<br><br>Thanks for your help and ideas!<br><br><br>John<br></div><br></div></body></html>