[ofw] OFED for Windows on 2008R2 SP1

Wed Apr 20 17:04:08 PDT 2011

Uninstall winOFED
Reboot
Run ibcleanup.bat
Reboot
Run ibscan; re-run ibcleanup if necessary.
Reboot
Now try your unattended install.

>-----Original Message-----
>From: Matthew Wallis [mailto:mattw at vpac.org]
>Sent: Wednesday, April 20, 2011 2:50 PM
>To: Smith, Stan
>Cc: ofw at lists.openfabrics.org
>Subject: Re: [ofw] OFED for Windows on 2008R2 SP1
>
>
>Hi Stan,
>             Meant to get back to this earlier, I have been able to confirm that the IPoIB driver is
>the OpenFabrics one. I have run the ibscan.bat and had a look at what that shows me, haven't used the
>IBCleanup as yet.
>
>Matt.
>
>On 16/04/2011, at 2:42 AM, Smith, Stan wrote:
>
>>> -----Original Message-----
>>> From: Matthew Wallis [mailto:mattw at vpac.org]
>>> Sent: Friday, April 15, 2011 2:17 AM
>>> To: Smith, Stan
>>> Cc: ofw at lists.openfabrics.org
>>> Subject: Re: [ofw] OFED for Windows on 2008R2 SP1
>>>
>>>
>>> Apparently we've found a solution for this. Of a sort.
>>>
>>> The problem appears to be that SP1 comes with a driver for the Mellanox card. If we remove that
>>> driver, the machines actually stop booting. Nice.
>>
>> I suspect you have not entirely removed the drivers; 3 .sys files: mlx4_bus.sys, mlx4_hca.sys and
>ipoib.sys.
>> Try the ibscan/ibcleanup mentioned in previous email prior to install.
>>
>>>
>>> Our current solution that obviously will require some more investigation is to install the official
>>> Mellanox drivers over the top of the Microsoft ones, then removing the Mellanox drivers. Once those
>>> are removed, the OFED drivers install without issue.
>>>
>>> That's the long way round.
>>>
>>> I can blacklist the driver in WSUS but I'm not sure I can stop SP1 from installing it.
>>
>> I do not recollect SP1 installing the HCA drivers proper, although Windows update did want to
>install the HCA drivers after the SP1 install/reboot.
>> BTW, the MS update drivers are Mellanox drivers which have been WHQL'ed; MS does not distribute MS
>developed HCA drivers at this time.
>>
>>>
>>> Matt.
>>>
>>> On 15/04/2011, at 9:52 AM, Matthew Wallis wrote:
>>>
>>>> <ua-install.log><Screen shot 2011-04-15 at 9.32.59 AM.png>
>>>>
>>>> Using the GUI, yes, IPoIB works fine. We've got two nodes running on GPFS from manual installs.
>>
>> I'm curious as to 'which' vendor's IPoIB you are running?
>> In windows\system32\drivers\ipoib.sys hold the mouse point over the filename and see what pops up?
>> You should see a company name 'OpenFabrics Alliance'.
>>
>>
>>> Looks like it's a MT26428
>>>> These are IBM branded cards with their own firmware on them. The Mellanox OEM and general drivers
>>> don't load at all on
>>>> these cards, we've had to push IBM to say they will support the OFED drivers.
>>
>>
>> You are in uncharted waters w.r.t. these IBM branded HCAs.
>> To the best of my understanding, winOFED HCA drivers (mlx4_bus/mlx4_hca) have never been loaded on
>IBM hardware.
>> Good to understand the GUI install worked.
>>
>> Stan.
>>
>>
>>>>
>>>> [root at m1-m ~]# ibstat
>>>> CA 'mlx4_0'
>>>> 	CA type: MT26428
>>>> 	Number of ports: 1
>>>> 	Firmware version: 2.7.700
>>>> 	Hardware version: b0
>>>> 	Node GUID: 0x0002c903000d8d02
>>>> 	System image GUID: 0x0002c903000d8d05
>>>> 	Port 1:
>>>> 		State: Active
>>>> 		Physical state: LinkUp
>>>> 		Rate: 40
>>>> 		Base lid: 1
>>>> 		LMC: 0
>>>> 		SM lid: 1
>>>> 		Capability mask: 0x0251086a
>>>> 		Port GUID: 0x0002c903000d8d03
>>>> 		Link layer: IB
>>>>
>>>>
>>>> <Screen shot 2011-04-15 at 9.47.27 AM.png>
>>>>
>>>> On 15/04/2011, at 1:49 AM, Smith, Stan wrote:
>>>>
>>>>> Which ConnectX HCA and which firmware version?
>>>>>
>>>>> Mellanox website has firmware burning tools if not latest firmware.
>>>>>
>>>>> Please try the following install cmd:  start/wait msiexec /i OFED_2-3_win7_x64.msi /qn /log
>c:\ua-
>>> install.log
>>>>>
>>>>> forward ua-install.log
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Stan.
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Matthew Wallis [mailto:mattw at vpac.org]
>>>>>> Sent: Wednesday, April 13, 2011 9:05 PM
>>>>>> To: Smith, Stan
>>>>>> Cc: ofw at lists.openfabrics.org
>>>>>> Subject: Re: [ofw] OFED for Windows on 2008R2 SP1
>>>>>>
>>>>>>
>>>>>> Just did the certificate based install using the msi, same results, only no prompting for certs
>>> this
>>>>>> time
>>>>>
>>>>> Progress.
>>>>>
>>>>>>
>>>>>> z:\support\OFED> rem-cert-add.bat OFED OpenFabrics.cer
>>>>>> Installing OpenFabrics.cer Cert on M1031
>>>>>>
>>>>>> M1031 SUCCESS: OFA TrustedPublisher cert installed
>>>>>>
>>>>>> z:\support\OFED>start/wait msiexec /i OFED_2-3_win7_x64.msi /qn
>>>>>>
>>>>>> z:\support\OFED>
>>>>>>
>>>>>> Had the usual flash message about the mlx_bus, but no IPoIB drivers, and vstat still complains
>>> about
>>>>>> missing complib.dll
>>>>>
>>>>> Since mlx4_bus does not appear to load correctly, then the mlx_hca driver will not be loaded,
>hence
>>> no complib.dll and vstat failure.
>>>>> In the past, HCA firmware problems have caused this type of behavior; although no current idea as
>>> to why a GUI  install would work while the unattended install fails?
>>>>>
>>>>> Speaking of a GUI install, does IPoIB configure correctly?
>>>>>
>>>>>>
>>>>>> Matt.
>>>>>>
>>>>>> On 14/04/2011, at 10:22 AM, Matthew Wallis wrote:
>>>>>>
>>>>>>> We're doing an unattended install of Server 2008R2 HPC edition from a network share, so I
>>>>>>> can either try to pre-seed that with the OFED drivers, or I can try and install the MSI via the
>>> HPC
>>>>>>> admin console. I'll give them a shot. I do need to make sure I get IPoIB as we're using GPFS
>for
>>>>>>> the filesystem.
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Matthew Wallis, HPC Systems Administrator
>>>>>> Victorian Partnership for Advanced Computing.
>>>>>> Ph: +61 3 9925 4452  Fax: +61 3 9925 4647
>>>>>
>>>>
>>>> --
>>>> Matthew Wallis, HPC Systems Administrator
>>>> Victorian Partnership for Advanced Computing.
>>>> Ph: +61 3 9925 4452  Fax: +61 3 9925 4647
>>>>
>>>> _______________________________________________
>>>> ofw mailing list
>>>> ofw at lists.openfabrics.org
>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
>>