[ofw] RE: HPC server 2008 IPoIB disrupts compute node Windows Update service?

[Smith, Stan]

Fab Tillier wrote:
> Hi Stan,
>
>> -----Original Message-----
>> From: Smith, Stan [mailto:stan.smith at intel.com]
>> Sent: Monday, August 03, 2009 9:52 AM
>>
>> Hi Fab,
>>   Do you have any theories on why/how IPoIB could disrupt Windows
>> update service?
>>
>> We have an HPC Server 2008 cluster configured such that compute
>> nodes, which are not directly connected to the enterprise
>> network/Internet, use the Private Ethernet (NAT enabled) to
>> communicate with the Head node which holds the enterprise/internet
>> Ethernet interface.
>> Once the compute nodes IE8 proxy is setup correctly, the compute
>> node's windows update service works fine (downloads & install OK).
>> Head-node Windows update service functions normally with IPoIB
>> enable or disabled.
>>
>> After a WinOF installation, IPoIB enabled and static IPv4 address set
>> on the compute node's IPoIB interface, the windows update service
>> just hangs when windows update is run.
>> It isn't until the IPoIB interface is disabled does the Windows
>> update service resume normal operation?
>>
>> What is it about IPoIB and HPC compute nodes?
>
> The OS isn't any different, so there shouldn't be anything special
> here.  However, if the Windows Update service tries to reach the
> internet using the IPoIB interface, it would likely fail since the
> IPoIB interface doesn't have NAT enabled for it.  I would have
> expected it to figure out the right interface to use though - are the
> subnet masks between the private Ethernet and IPoIB subnets distinct?

I too would have thought the network stack would figure it out...?

Distinct subnets Private @ [192.168.10.x] IPoIB @ [192.168.1.x} although the netmask is the same 255.255.255.0.
What values do you use for your NETMASKs on cluster interfaces?

Interface ordering could be a problem.
I'll try reordering the interface access order from IPoIB, Private, Enterprise --> Private, Enterprise, IPoIB.
Per chance do you know of a cmd line utility which does this in lieu of running the Network Interface Mgmt GUI?

>
> If you uninstall the HPC Pack from the compute node, does the problem
> persist or go away?

Have not tried this larger hammer; yet.

>
> A netmon trace for the outbound traffic for both the IPoIB and
> non-IPoIB cases would likely shed a lot of light on the issue.
>
>> We do not see the Windows update service problems when WinOF/IPoIB is
>> installed on vanilla server 2008 or 2008/R2?
>
> Do you also have the NAT in place in this scenario, on only the
> Ethernet network and not IPoIB?

No NAT in this case.

>
> -Fab