[ofw] Windows SRP perf question

[Smith, Stan]

From: ofw-bounces at lists.openfabrics.org [mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Todd Pisek
Sent: Wednesday, October 02, 2013 10:34 AM
To: ofw at lists.openfabrics.org
Subject: [ofw] Windows SRP perf question

Greetings,

I'm not new to Infiniband, but I am new to Windows.

I'm running the 3.2 OFW release on Windows 2008R2 using a Dell R720 with dual sandy bridge 4-core processors. The HCA is a Mellanox ConnectX3-vpi. The application is a video stream simulator running atop our file system which is a kernel file system filter driver.  The experiment is accessing block storage using SRP.

As the number of parallel video streams is increased, the bandwidth flattens. It appears the issue is saturation of one cpu that is processing all the SRP completion dpc's. Looking at the OFED source, I believe the reason is that srp uses one sq/qp/rq per target. I believe this means that all response completion event dpc's will be queued to the cpu associated with the target's  rq cq.

One solution I'd like to try is to put the requestor's cpu number in the MSB of the command tag and then use that information in __srp_process_session_recv_completion  to obtain the cpu of the initiator and then construct a dpc targeted to the initiating cpu to finish the completion processing (call __srp_process_recv_completion)

I have 3 questions:

1) Is my analysis correct, that is, are all srp responses from the same target processed by the same cpu ?
2) Does my solution seem reasonable?
3) How do I build the OFW code?

Hello,
  A couple of points along the way.
SRP initiator has been without support for a couple of years.
Your analysis sounds reasonable for what little I know of SRP internals, although worth an experiment.
Download the Windows 7 WDK (Windows Driver Kit) 7600.16385.1 ; install WDK ( you should see C:\WDK\7600.16385.1)
>From a command window:

cd <top of OFED src tree>

#Following setup the build env
set _DDK=%systemDrive%\WInDDK\7600.16385.1

%_DDK%\bin\setenv.bat %_DDK% fre x64 WIN7 no_oacr

#following may not be required ?
Set PLATFORM_SDK_PATH=C:\PROGRA~1\MICROS~3\Windows\v6.1
Set ND_SDK_PATH=C:\PROGRA~1\MI1DC7C~1\NetDIrect

cd core\
build
cd ..\ulp\srp
build
obj/dll files in ..\..\bin\kernel\...

OFED 3.2 is the last OFED for Windows release; see Mellanox WinOF distribution - you may be able to build the SRP from their distribution?
Why the last, Microsoft has released NetworkDirect as their RDMA API; network vendors are moving towards supporting NetworkDirect which alleviates need for OFED for Windows.

Stan.

Thanks.

--- Todd

________________________________
The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ofw/attachments/20131002/bc2a831b/attachment.html>