[ofw] SRP problems with WinOF 1.0.1

Tue May 13 14:22:27 PDT 2008

we spent some time doing a little benchmarking on windows last week
and discovered some things about the behavior of the srp miniport
(storport) driver and windows.

windows server 2003 (sp2) 32bit running on 64-bit hardware uses 16
NumberOfPhysicalBreaks regardless of what the srp miniport tries to force.
if you go one step further and try to emulate a 32-bit system as much
as possible by disabled PAE and DEP, the NumberOfPhysicalBreaks is then
limited to 8.  This applies to the 32-bit version of windows server 2008
as well and likely applies to any of the 32-bit versions of windows
running on 64-bit hardware.  we were unable to test a 32-bit version
of windows running on an actual 32-bit processor.

the story changes for 64-bit windows.  windows xp, windows server 2003
and windows server 2008 (the 64-bit versions) will respect the srp
miniport's increase in NumberOfPhysicalBreaks.  using any of these
operating systems we were able get 1024k i/o's. 

however, if you try to stripe several disks together, the i/o drops to
64k again.  this is likely because the default stripe size for windows is
64k and the window's i/o scheduler doesnt seem to aggregate pending i/o's.
we were unable to test larger stripe sizes since we dont have a utility
to create anything other than the default stripe size.

it is difficult to understand why the 32-bit version of windows is
using the 16 limit.  i am guessing it has something to do with the
IOGetDmaAdapter() for the virtual/internal device associated with the
srp miniport.  apparently 32-bit windows seems to have a hard limit for
the available map registers in this case.  additionally, if you dont
allow the srp miniport to upgrade NumberOfPhysicalBreaks on 64-bit
windows, the i/o rates are still about 60% faster on 64-bit windows.
this makes me think that the 32-bit version of windows might be using
a double/bounce buffer?

at the end of this testing, we were able to get 4 threads on iometer
running across 4 luns (one per controller) for a total of 1.5GB/s reads
on our ddn s2a9500.  the windows host had a single ddr adapter and the
srp miniport was tuned to force 1024k i/o sizes.

In message <120DDDEC0C4AA045B93BDB63FE45905E0A600D at EPEXCH1.qlogic.org>,"Eleanor
 Witiak" writes:
>This is a multi-part message in MIME format.
>
>--===============1373268742==
>Content-class: urn:content-classes:message
>Content-Type: multipart/alternative;
>	boundary="----_=_NextPart_001_01C869C2.4507D9FA"
>
>This is a multi-part message in MIME format.
>
>------_=_NextPart_001_01C869C2.4507D9FA
>Content-Type: text/plain;
>	charset="iso-8859-1"
>Content-Transfer-Encoding: quoted-printable
>
>Hi Fab,
>=20
>What would you like me to try?
>=20
>Eleanor
>
>________________________________
>
>From: ofw-bounces at lists.openfabrics.org on behalf of Fab Tillier
>Sent: Thu 2/7/2008 2:29 PM
>To: Usha Srinivasan; Jan Bottorff; ofw at lists.openfabrics.org
>Subject: RE: [ofw] SRP problems with WinOF 1.0.1
>
>
>
>Hi Folks,
>
>I'm trying to find things out on my end.  I don't have any hardware to =
>test things, and would like to have a volunteer for trying things out.
>
>-Fab
>
>-----Original Message-----
>From: ofw-bounces at lists.openfabrics.org =
>[mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Usha Srinivasan
>Sent: Thursday, February 07, 2008 5:50 AM
>To: Jan Bottorff; ofw at lists.openfabrics.org
>Subject: RE: [ofw] SRP problems with WinOF 1.0.1
>
>Can someone at Microsoft confirm that StorPort driver in SP2 indeed
>ignores a miniport's attempt to increase the NumberOfPhysicalBreaks
>above 17?
>
>We are running WinOF SRP with SP2 and are trying to get buffer sizes
>above 64K bytes, and it is just not happening. We cannot switch back to
>SP1.
>
>If this is simply not doable in SP2 with a Storport Miniport, we would
>sure like to know, so we can drop this and move on.
>
>Fab, can you possibly get a confirmation on this????
>
>Thanks.
>Usha
>
>-----Original Message-----
>From: ofw-bounces at lists.openfabrics.org
>[mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Jan Bottorff
>Sent: Wednesday, February 06, 2008 7:30 PM
>To: ofw at lists.openfabrics.org
>Subject: RE: [ofw] SRP problems with WinOF 1.0.1
>
>> Seems like MS patched the StorPort in SP2.
>
>SP2 has an updated storport driver. There is also a post SP2 update to
>the storport driver (from about April/May 2007), available from I
>believe the Microsoft support website. My unconfirmed guess is we may
>eventually see an updated W2K3 storport with some of the W2K8 changes.
>
>There are changes in both the SP2 and post-SP2 storport that can affect
>storage miniport drivers, like the SRP driver.
>
>Jan
>
>
>_______________________________________________
>ofw mailing list
>ofw at lists.openfabrics.org
>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
>_______________________________________________
>ofw mailing list
>ofw at lists.openfabrics.org
>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
>_______________________________________________
>ofw mailing list
>ofw at lists.openfabrics.org
>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
>
>
>
>------_=_NextPart_001_01C869C2.4507D9FA
>Content-Type: text/html;
>	charset="iso-8859-1"
>Content-Transfer-Encoding: quoted-printable
>
><HTML dir=3Dltr><HEAD><TITLE>https://qlm.qlogic.com - RE: [ofw] SRP =
>problems with WinOF 1.0.1</TITLE>=0A=
><META http-equiv=3DContent-Type content=3D"text/html; charset=3Dunicode">=0A=
><META content=3D"MSHTML 6.00.3790.186" name=3DGENERATOR></HEAD>=0A=
><BODY>=0A=
><DIV id=3DidOWAReplyText1703 dir=3Dltr>=0A=
><DIV dir=3Dltr><FONT face=3DArial color=3D#000000 size=3D2>Hi =
>Fab,</FONT></DIV>=0A=
><DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT> </DIV>=0A=
><DIV dir=3Dltr><FONT face=3DArial size=3D2>What would you like me to =
>try?</FONT></DIV>=0A=
><DIV dir=3Dltr><FONT face=3DArial size=3D2></FONT> </DIV>=0A=
><DIV dir=3Dltr><FONT face=3DArial size=3D2>Eleanor</FONT></DIV></DIV>=0A=
><DIV dir=3Dltr><BR>=0A=
><HR tabIndex=3D-1>=0A=
><FONT face=3DTahoma size=3D2><B>From:</B> =
>ofw-bounces at lists.openfabrics.org on behalf of Fab =
>Tillier<BR><B>Sent:</B> Thu 2/7/2008 2:29 PM<BR><B>To:</B> Usha =
>Srinivasan; Jan Bottorff; ofw at lists.openfabrics.org<BR><B>Subject:</B> =
>RE: [ofw] SRP problems with WinOF 1.0.1<BR></FONT><BR></DIV>=0A=
><DIV>=0A=
><P><FONT size=3D2>Hi Folks,<BR><BR>I'm trying to find things out on my =
>end.  I don't have any hardware to test things, and would like to =
>have a volunteer for trying things out.<BR><BR>-Fab<BR><BR>-----Original =
>Message-----<BR>From: ofw-bounces at lists.openfabrics.org [<A =
>href=3D"mailto:ofw-bounces at lists.openfabrics.org">mailto:ofw-bounces at list=
>s.openfabrics.org</A>] On Behalf Of Usha Srinivasan<BR>Sent: Thursday, =
>February 07, 2008 5:50 AM<BR>To: Jan Bottorff; =
>ofw at lists.openfabrics.org<BR>Subject: RE: [ofw] SRP problems with WinOF =
>1.0.1<BR><BR>Can someone at Microsoft confirm that StorPort driver in =
>SP2 indeed<BR>ignores a miniport's attempt to increase the =
>NumberOfPhysicalBreaks<BR>above 17?<BR><BR>We are running WinOF SRP with =
>SP2 and are trying to get buffer sizes<BR>above 64K bytes, and it is =
>just not happening. We cannot switch back to<BR>SP1.<BR><BR>If this is =
>simply not doable in SP2 with a Storport Miniport, we would<BR>sure like =
>to know, so we can drop this and move on.<BR><BR>Fab, can you possibly =
>get a confirmation on =
>this????<BR><BR>Thanks.<BR>Usha<BR><BR>-----Original =
>Message-----<BR>From: ofw-bounces at lists.openfabrics.org<BR>[<A =
>href=3D"mailto:ofw-bounces at lists.openfabrics.org">mailto:ofw-bounces at list=
>s.openfabrics.org</A>] On Behalf Of Jan Bottorff<BR>Sent: Wednesday, =
>February 06, 2008 7:30 PM<BR>To: ofw at lists.openfabrics.org<BR>Subject: =
>RE: [ofw] SRP problems with WinOF 1.0.1<BR><BR>> Seems like MS =
>patched the StorPort in SP2.<BR><BR>SP2 has an updated storport driver. =
>There is also a post SP2 update to<BR>the storport driver (from about =
>April/May 2007), available from I<BR>believe the Microsoft support =
>website. My unconfirmed guess is we may<BR>eventually see an updated =
>W2K3 storport with some of the W2K8 changes.<BR><BR>There are changes in =
>both the SP2 and post-SP2 storport that can affect<BR>storage miniport =
>drivers, like the SRP =
>driver.<BR><BR>Jan<BR><BR><BR>___________________________________________=
>____<BR>ofw mailing list<BR>ofw at lists.openfabrics.org<BR><A =
>href=3D"http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw">http:/=
>/lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw</A><BR>______________=
>_________________________________<BR>ofw mailing =
>list<BR>ofw at lists.openfabrics.org<BR><A =
>href=3D"http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw">http:/=
>/lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw</A><BR>______________=
>_________________________________<BR>ofw mailing =
>list<BR>ofw at lists.openfabrics.org<BR><A =
>href=3D"http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw">http:/=
>/lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw</A><BR></FONT></P></D=
>IV></BODY></HTML>
>------_=_NextPart_001_01C869C2.4507D9FA--
>
>--===============1373268742==
>Content-Type: text/plain; charset="us-ascii"
>MIME-Version: 1.0
>Content-Transfer-Encoding: 7bit
>Content-Disposition: inline
>
>_______________________________________________
>ofw mailing list
>ofw at lists.openfabrics.org
>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
>--===============1373268742==--
>