[ofa-general] OFED interfering with Ethernet

[Sebastian Kalcher]

Hi all,

I need a little direction to cope with a strange problem:

I have node A and B connected via an HP Procurve Gigabit Ethernet  
switch. Node B has also an Mellanox MT26428 QDR HCA.

I run iperf to test the GbE connection between A and B. Everything works fine.

If I load the openib drivers (i.e. "/etc/init.d/openib start") on B.  
The iperf connection gets very flaky. Sometimes it stalls for several  
seconds (or tens of seconds) and then starts again, after a while it  
completely freezes. I don't see any TCP timeouts, the iperf processes  
are simply sleeping, and no packets are transferred. It doesn't matter  
if the ib0 device is configured or not, or whether opensm is running.

If I do an "openib stop" and restart iperf everything returns to normal.

The behavior is reproducible on different nodes with the same  
constellation. I tried OFED 1.4 and 1.5beta (kernel 2.6.24). What  
completely confuses me here is that it is the ethernet connection that  
gets screwed up, IB isn't used at all other than that the drivers are  
loaded.

Any hint where to continue would be greatly appreciated.

Sebastian