[ofw] OFED for Windows on 2008R2 SP1

Wed Apr 20 14:49:58 PDT 2011

Hi Stan,
             Meant to get back to this earlier, I have been able to confirm that the IPoIB driver is the OpenFabrics one. I have run the ibscan.bat and had a look at what that shows me, haven't used the IBCleanup as yet.

Matt.

On 16/04/2011, at 2:42 AM, Smith, Stan wrote:

>> -----Original Message-----
>> From: Matthew Wallis [mailto:mattw at vpac.org]
>> Sent: Friday, April 15, 2011 2:17 AM
>> To: Smith, Stan
>> Cc: ofw at lists.openfabrics.org
>> Subject: Re: [ofw] OFED for Windows on 2008R2 SP1
>> 
>> 
>> Apparently we've found a solution for this. Of a sort.
>> 
>> The problem appears to be that SP1 comes with a driver for the Mellanox card. If we remove that
>> driver, the machines actually stop booting. Nice.
> 
> I suspect you have not entirely removed the drivers; 3 .sys files: mlx4_bus.sys, mlx4_hca.sys and ipoib.sys.
> Try the ibscan/ibcleanup mentioned in previous email prior to install.
> 
>> 
>> Our current solution that obviously will require some more investigation is to install the official
>> Mellanox drivers over the top of the Microsoft ones, then removing the Mellanox drivers. Once those
>> are removed, the OFED drivers install without issue.
>> 
>> That's the long way round.
>> 
>> I can blacklist the driver in WSUS but I'm not sure I can stop SP1 from installing it.
> 
> I do not recollect SP1 installing the HCA drivers proper, although Windows update did want to install the HCA drivers after the SP1 install/reboot.
> BTW, the MS update drivers are Mellanox drivers which have been WHQL'ed; MS does not distribute MS developed HCA drivers at this time.
> 
>> 
>> Matt.
>> 
>> On 15/04/2011, at 9:52 AM, Matthew Wallis wrote:
>> 
>>> <ua-install.log><Screen shot 2011-04-15 at 9.32.59 AM.png>
>>> 
>>> Using the GUI, yes, IPoIB works fine. We've got two nodes running on GPFS from manual installs.
> 
> I'm curious as to 'which' vendor's IPoIB you are running?
> In windows\system32\drivers\ipoib.sys hold the mouse point over the filename and see what pops up?
> You should see a company name 'OpenFabrics Alliance'.
> 
> 
>> Looks like it's a MT26428
>>> These are IBM branded cards with their own firmware on them. The Mellanox OEM and general drivers
>> don't load at all on
>>> these cards, we've had to push IBM to say they will support the OFED drivers.
> 
> 
> You are in uncharted waters w.r.t. these IBM branded HCAs.
> To the best of my understanding, winOFED HCA drivers (mlx4_bus/mlx4_hca) have never been loaded on IBM hardware.
> Good to understand the GUI install worked.
> 
> Stan.
> 
> 
>>> 
>>> [root at m1-m ~]# ibstat
>>> CA 'mlx4_0'
>>> 	CA type: MT26428
>>> 	Number of ports: 1
>>> 	Firmware version: 2.7.700
>>> 	Hardware version: b0
>>> 	Node GUID: 0x0002c903000d8d02
>>> 	System image GUID: 0x0002c903000d8d05
>>> 	Port 1:
>>> 		State: Active
>>> 		Physical state: LinkUp
>>> 		Rate: 40
>>> 		Base lid: 1
>>> 		LMC: 0
>>> 		SM lid: 1
>>> 		Capability mask: 0x0251086a
>>> 		Port GUID: 0x0002c903000d8d03
>>> 		Link layer: IB
>>> 
>>> 
>>> <Screen shot 2011-04-15 at 9.47.27 AM.png>
>>> 
>>> On 15/04/2011, at 1:49 AM, Smith, Stan wrote:
>>> 
>>>> Which ConnectX HCA and which firmware version?
>>>> 
>>>> Mellanox website has firmware burning tools if not latest firmware.
>>>> 
>>>> Please try the following install cmd:  start/wait msiexec /i OFED_2-3_win7_x64.msi /qn /log c:\ua-
>> install.log
>>>> 
>>>> forward ua-install.log
>>>> 
>>>> Thanks,
>>>> 
>>>> Stan.
>>>> 
>>>>> -----Original Message-----
>>>>> From: Matthew Wallis [mailto:mattw at vpac.org]
>>>>> Sent: Wednesday, April 13, 2011 9:05 PM
>>>>> To: Smith, Stan
>>>>> Cc: ofw at lists.openfabrics.org
>>>>> Subject: Re: [ofw] OFED for Windows on 2008R2 SP1
>>>>> 
>>>>> 
>>>>> Just did the certificate based install using the msi, same results, only no prompting for certs
>> this
>>>>> time
>>>> 
>>>> Progress.
>>>> 
>>>>> 
>>>>> z:\support\OFED> rem-cert-add.bat OFED OpenFabrics.cer
>>>>> Installing OpenFabrics.cer Cert on M1031
>>>>> 
>>>>> M1031 SUCCESS: OFA TrustedPublisher cert installed
>>>>> 
>>>>> z:\support\OFED>start/wait msiexec /i OFED_2-3_win7_x64.msi /qn
>>>>> 
>>>>> z:\support\OFED>
>>>>> 
>>>>> Had the usual flash message about the mlx_bus, but no IPoIB drivers, and vstat still complains
>> about
>>>>> missing complib.dll
>>>> 
>>>> Since mlx4_bus does not appear to load correctly, then the mlx_hca driver will not be loaded, hence
>> no complib.dll and vstat failure.
>>>> In the past, HCA firmware problems have caused this type of behavior; although no current idea as
>> to why a GUI  install would work while the unattended install fails?
>>>> 
>>>> Speaking of a GUI install, does IPoIB configure correctly?
>>>> 
>>>>> 
>>>>> Matt.
>>>>> 
>>>>> On 14/04/2011, at 10:22 AM, Matthew Wallis wrote:
>>>>> 
>>>>>> We're doing an unattended install of Server 2008R2 HPC edition from a network share, so I
>>>>>> can either try to pre-seed that with the OFED drivers, or I can try and install the MSI via the
>> HPC
>>>>>> admin console. I'll give them a shot. I do need to make sure I get IPoIB as we're using GPFS for
>>>>>> the filesystem.
>>>>>> 
>>>>> 
>>>>> --
>>>>> Matthew Wallis, HPC Systems Administrator
>>>>> Victorian Partnership for Advanced Computing.
>>>>> Ph: +61 3 9925 4452  Fax: +61 3 9925 4647
>>>> 
>>> 
>>> --
>>> Matthew Wallis, HPC Systems Administrator
>>> Victorian Partnership for Advanced Computing.
>>> Ph: +61 3 9925 4452  Fax: +61 3 9925 4647
>>> 
>>> _______________________________________________
>>> ofw mailing list
>>> ofw at lists.openfabrics.org
>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
> 