[ofw] some WinOF questions

[Tzachi Dar]

Hi Diego,

Please note that there is a bug on the way the buffer sizes between
windows and Linux work.
In short this means that windows starts with big buffers and never
changes them. On Linux on the other hand, they start with small buffers
and increase them. When windows and Linux work together, this ends with
the connection using small buffers and therefore low BW (~3000 Mb/sec).

There are two workarounds for this problem:

1) Changing the code on Linux, please do the following:
It seems that the root of the "problem" is on line 180 of the file
sdp_cma.c There is a function called sdp_init_qp() and there is a line:

	sdp_sk(sk)->recv_frags = 0;

This line tells the Linux side to start with smaller packets. If they
change this line to be 

sdp_sk(sk)->recv_frags = 8; 

Than Linux will start with bigger packets and (8*4k + 2k =34kb) and they
can also get high performance without changing anything at the windows
side. Please note that without changing anything at the windows side,
the packet size will default to 16kb which will somewhat hurt
performance.

2) Changing the code on windows:
Attached is the windows driver that has a work around for the problem.
It does two things:
	1) Increase the windows buffers to 32kb.
	2) fools the Linux side to be using large buffers. This should
allow the Linux driver to send bigger packets without any changes, and
should allow them to reach near wire speed BW.
To run their test, there is no need to do anything beside copying the
file to the windows\system32\drivers directory, and restarting SDP. 

Two more things to note:
1) you can get to a better BW by using windows XP 64 bits. It's price
should be similar to the XP 32.
2) The fixes that I have sent are meant to increase the BW when Linux is
sending to windows. If you want BW to increase on the other side
(windows to Linux) you should probably apply both fixes.

Thanks
Tzachi

> -----Original Message-----
> From: Diego Guella [mailto:diego.guella at sircomtech.com] 
> Sent: Thursday, November 08, 2007 9:39 AM
> To: Smith, Stan; Tzachi Dar; Sufficool, Stanley; Hefty, Sean
> Cc: ofw at lists.openfabrics.org
> Subject: Re: [ofw] some WinOF questions
> 
> 
> ----- Original Message ----- 
> >From: "Smith, Stan" <stan.smith at intel.com> Microsoft 
> requires those who 
> >distribute SDP to sign a license with MS concerning IP rights of SDP.
> >Since WinOF is open-source MS has declined to offer an SDP licensing 
> >option, therefore SDP is not part of WinOF (Windows 
> OpenFabrics which 
> >is based on the openib-windows src tree).
> >
> >On the other hand, Mellanox has signed a licensing agreement with MS 
> >and therefore distributes SDP as part of their WinIB stack which is 
> >also based on the openIB-windows src plus Mellanox value add.
> 
> Oh, licensing problems. So I'll try with Mellanox WinIB 1.3
> 
> >The WinOF answer to missing SDP is WSD (Winsock Direct 
> Provider) which 
> >from the user's perspective provides similar utilization of 
> InfiniBand 
> >hardware as does SDP.
> 
> I think we can't use WSD.
> 1) Requires Windows Server 2003, wich means too many Euros (or $).
> 2) Can't communicate with Linux using WSD
> 
> 
> >Within the Linux world, connected mode IPoIB approaches or in some 
> >cases surpasses SDP performance.
> >Connected mode IPoIB is appearing on the WinOF 2008 horizon.
> 
> If it means early 2008, cool! Otherwise I think I need to 
> find another solution
> 
> 
> 
> Thanks,
> Diego
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Sdp.sy_
Type: application/octet-stream
Size: 74752 bytes
Desc: Sdp.sy_
URL: <http://lists.openfabrics.org/pipermail/ofw/attachments/20071108/02e24acc/attachment.obj>