[Users] IPoIB not working on Windows 2008 r2 - need help

Coulter, Susan K skc at lanl.gov
Thu Jun 6 14:31:29 PDT 2013 

I am not a Windows person, but if ipconfig shows the IPoIB interface as up and running, then this seems like a routing problem on the node itself.

The fact that the ICMP packets are being sent out a different NIC tells me the host has no route to the IPoIB space and is using a default route to send the packet out the 'default' interface.



On Jun 6, 2013, at 1:32 PM, Orion Poplawski <orion at cora.nwra.com> wrote:

> I'm trying for the first time to get IPoIB working on one of our Windows servers.  The network is working fine between some Linux machines.  Details:
> 
> InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA]
> Windows Server 2008 r2
> MLNX_WinOF_VPI_2_1_2_win7_x64.msi (as recommended by the mellanox download page for InfiniHost III adapters)
> 
> I don't notice any errors, the adapter shows up fine and I can configure it with a static IP address.  After configuring it (or after boot) I can ping it from another machine for about 10 seconds before it stops responding.  When I ping out from the machine at this point, the icmp packets are being sent out the main ethernet interface (which is a different IP network) and I can see them get to our router.  ibdiagnet does not report any errors.  ipconfig and netstart -r seem fine.
> 
> I see the following in my opensm log:
> 
> Jun 06 11:51:50 600282 [29FC1700] 0x02 -> log_notice: Reporting Generic Notice type:1 num:128 (Link state change) from LID:2 GID:fe80::1:6:6a00:d800:242
> Jun 06 11:51:51 011771 [1F5B0700] 0x02 -> osm_ucast_mgr_process: minhop tables configured on all switches
> Jun 06 11:51:51 016889 [1F5B0700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:64 (GID in service) from LID:1 GID:fe80::1:5:ad00:c:5ced
> Jun 06 11:51:51 016899 [1F5B0700] 0x02 -> state_mgr_report_new_ports: Discovered new port with GUID:0x0005ad00000c5ced LID range [16,16] of node: Topspin DDR-HCAe LX x8
> Jun 06 11:51:51 027491 [1F5B0700] 0x02 -> SUBNET UP
> Jun 06 11:51:56 333829 [213B3700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:1 GID:ff12:1405:ffff::3333:0:1
> Jun 06 11:51:56 333875 [209B2700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:1 GID:ff12:1405:ffff::3333:ff76:9ac6
> Jun 06 11:51:56 603980 [295C0700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:1 GID:ff12:1405:ffff::3333:0:2
> Jun 06 11:51:56 604270 [245B8700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:1 GID:ff12:1405:ffff::3333:0:16
> Jun 06 11:52:15 854497 [263BB700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:1 GID:ff12:1405:ffff::3333:1:2
> Jun 06 11:52:15 857261 [213B3700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:1 GID:ff12:1405:ffff::3333:1:3
> Jun 06 11:52:15 857968 [209B2700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:1 GID:ff12:401b:ffff::16
> Jun 06 11:52:15 963577 [21DB4700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:1 GID:ff12:401b:ffff::fc
> Jun 06 12:04:56 535293 [26DBC700] 0x01 -> mcmr_rcv_leave_mgrp: ERR 1B25: Received an invalid delete request for MGID: ff12:1405:ffff::3333:0:1 for PortGID: fe80::5:ad00:c:5ced
> Jun 06 12:04:56 535870 [277BD700] 0x01 -> mcmr_rcv_leave_mgrp: ERR 1B25: Received an invalid delete request for MGID: ff12:1405:ffff::3333:1:3 for PortGID: fe80::5:ad00:c:5ced
> Jun 06 12:04:56 535908 [259BA700] 0x01 -> mcmr_rcv_leave_mgrp: ERR 1B25: Received an invalid delete request for MGID: ff12:1405:ffff::3333:1:2 for PortGID: fe80::5:ad00:c:5ced
> Jun 06 12:04:56 535942 [23BB7700] 0x01 -> mcmr_rcv_leave_mgrp: ERR 1B25: Received an invalid delete request for MGID: ff12:401b:ffff::fc for PortGID: fe80::5:ad00:c:5ced
> Jun 06 12:04:56 535970 [281BE700] 0x01 -> mcmr_rcv_leave_mgrp: ERR 1B25: Received an invalid delete request for MGID: ff12:401b:ffff::16 for PortGID: fe80::5:ad00:c:5ced
> Jun 06 12:04:56 536014 [277BD700] 0x01 -> mcmr_rcv_leave_mgrp: ERR 1B25: Received an invalid delete request for MGID: ff12:1405:ffff::3333:ff76:9ac6 for PortGID: fe80::5:ad00:c:5ced
> Jun 06 12:04:56 536042 [209B2700] 0x01 -> mcmr_rcv_leave_mgrp: ERR 1B25: Received an invalid delete request for MGID: ff12:1405:ffff::3333:0:16 for PortGID: fe80::5:ad00:c:5ced
> Jun 06 12:04:56 536634 [227B5700] 0x01 -> mcmr_rcv_leave_mgrp: ERR 1B25: Received an invalid delete request for MGID: ff12:1405:ffff::3333:0:2 for PortGID: fe80::5:ad00:c:5ced
> Jun 06 12:06:29 959894 [295C0700] 0x01 -> mcmr_rcv_join_mgrp: ERR 1B13: validate_modify failed from port 0x0005ad00000c5ced (Topspin DDR-HCAe LX x8), sending IB_SA_MAD_STATUS_REQ_INVALID
> Jun 06 12:06:29 960518 [231B6700] 0x01 -> mcmr_rcv_leave_mgrp: ERR 1B25: Received an invalid delete request for MGID: ff12:1405:ffff::3333:1:2 for PortGID: fe80::5:ad00:c:5ced
> Jun 06 12:06:36 629355 [26DBC700] 0x01 -> mcmr_rcv_leave_mgrp: ERR 1B25: Received an invalid delete request for MGID: ff12:401b:ffff::1 for PortGID: fe80::5:ad00:c:5ced
> Jun 06 12:06:36 629416 [259BA700] 0x01 -> mcmr_rcv_leave_mgrp: ERR 1B25: Received an invalid delete request for MGID: ff12:401b:ffff::ffff:ffff for PortGID: fe80::5:ad00:c:5ced
> Jun 06 12:06:36 638659 [21DB4700] 0x01 -> mcmr_rcv_join_mgrp: ERR 1B13: validate_modify failed from port 0x0005ad00000c5ced (Topspin DDR-HCAe LX x8), sending IB_SA_MAD_STATUS_REQ_INVALID
> Jun 06 12:06:36 638853 [245B8700] 0x01 -> mcmr_rcv_leave_mgrp: ERR 1B25: Received an invalid delete request for MGID: ff12:401b:ffff::ffff:ffff for PortGID: fe80::5:ad00:c:5ced
> 
> This last message repeats quite a bit within that second, and then stops.
> 
> Any ideas?
> 
> -- 
> Orion Poplawski
> Technical Manager                     303-415-9701 x222
> NWRA, Boulder/CoRA Office             FAX: 303-415-9702
> 3380 Mitchell Lane                       orion at nwra.com
> Boulder, CO 80301                   http://www.nwra.com
> _______________________________________________
> Users mailing list
> Users at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users

====================================

Susan Coulter
HPC-3 Network/Infrastructure
505-667-8425
Increase the Peace...
An eye for an eye leaves the whole world blind
====================================

More information about the Users mailing list