> ib_mthca 0000:02:00.0: Catastrophic error detected: internal error This means your HCA detected an internal error -- overheating, power glitch, cosmic ray, firmware bug, something like that.