[Users] HP BLc QLogic 4X QDR IB Switch oddness

[John Valdes]

Andrei,

On Fri, May 24, 2013 at 01:51:00PM +0100, Andrei Mikhailovsky wrote:
> 
> I was wondering if anyone on the mailing list has experienced an odd behaviour with HP BLc QLogic 4X QDR IB Switch (505958-B21)? [...]
> 
> My problem is that every time I reboot the blade servers which are connected to the above blade switch the switch ports take ages to Activate. [...]
> 
> However, the server which are connected to the same switch with a cable (not interconnected blade servers) do not have this issue and get port state Active in seconds every time. 

This last fact, that servers directly attached to the switch w/ a
cable negotiate link immediately, I think clearly points to the blade
servers and/or blade chassis as the source of the problem and not the
switch. 

What's the physical topology of the IB network between the blade
servers and the switch?  I'm not familiar w/ the HP BLc, but I'm
guessing each blade server has an internal IB Mezzanine HCA which
connects to a chassis backplane, correct?  How does the IB from the
backplane connect to the switch?  Is there specialized connector
between them, or does it use a standard IB cable (possibly more than
one)?  Does the chassis backplane have an IB switch chip?

Some random thoughts/questions:

1) Where is the subnet manager running?  What routing protocol is it
   using (eg, minhop, updown, etc)?

2) When rebooting the blade servers, are you rebooting all the ones in
   the same chassis at the same time?  If you stagger the reboot of
   each server, or if you just reboot one server, does it still take a
   long time for it to negotiate link?

3) The problem could be a firmware issue; since the directly attahed
   servers and the blade servers must have different HCAs (the former
   standard PCI HCAs and the latter mezzanine HCAs I'm assuming), that
   suggests they'll have different firmware.  Unforutnately, you say
   you're running the latest firmware already.

John