[Users] connect Mellanox IB FDR adapter to QLogic IB QDR switch?

[Marko Kobal]

Hi,

Well, I noticed that the links did eventually came up at the second node 
as well ... however it took several hours to do the link initialization:

cat /var/log/messages | grep mlx4_core

node_1:
Jun  6 17:42:08 node_1 kernel: mlx4_core 0000:10:00.0: PCI INT A -> GSI 
30 (level, low) -> IRQ 30
Jun  7 04:35:33 node_1 kernel: mlx4_core 0000:10:00.0: mlx4_ib: Port 1 
logical link is up

node_2:
Jun 12 16:05:12 node_2 kernel: mlx4_core 0000:10:00.0: PCI INT A -> GSI 
30 (level, low) -> IRQ 30
Jun 12 20:50:24 node_2 kernel: mlx4_core 0000:10:00.0: mlx4_ib: Port 1 
logical link is up

The connection is now actually working, however the extremely long time 
to make the link initialization is really weird (and annoying when doing 
server or openibd restart) ... Anyone seen this kind of behaviour on 
Mellanox adapters?


Kind regards, Marko Kobal


On 12. 06. 2013 17:12, Marko Kobal wrote:
> Hi,
>
> I have a IBM iDataPlex cluster with QLogic IB QDR switches (QLogic 12200
> (iDPx) 36-port QDR InfiniBand Switch), currently there are servers with
> QLogic QME7342 QDR HCA adapters connected to those IB switches ... all
> is playing nice.
>
> Recently I've got a couple of Mellanox IB FDR adapters (MT27500 Family
> [ConnectX-3]) and I would like them play nicely with existing IB network
> since I would like to do some research with Mellanox adapters.
>
> On last Thursday (5 days ago) I've installed one FDR Mellanox adapter in
> one of the servers (replaced QLogic QDR adapter) and connected it to the
> QDR QLogic IB switch. Other servers with QDR QLogic adapters are also
> connected to the switch and are playing nicely. Mellanox FDR Adapter was
> detected by the server and OS (centos6), latest ofed installed. After
> reboot IB diag tools showed link is down, adapter's leds were off ... I
> didn't have time to play with it so I left it as it was. Yesterday I
> checked the status once again and I found out that the link is now
> (State: Active; Physical state: LinkUp; Rate: 40) ... so this looks
> fine. Today I went ahead and installed adapters in additional two
> servers and I see the same situation - link is down, adapter's leds were
> off ...
>
> Is it possible that link initialization is taking sooo long (it is now
> more then one hour since I connected the adapters with the switch)? Is
> there any "magic" to force the initialization/connection or to persuade
> Mellanox FDR adapter to go directly to QDR speed?
>
> Has anybody got any experiences in mixing QDR and FDR IB, especially
> connecting Mellanox FDR adapters to QLogic QDR IB switches? Thanks for help!
>
>