[ofw] some WinOF questions

[Tzachi Dar]

See bellow.

Thanks
Tzachi 

> -----Original Message-----
> From: Smith, Stan [mailto:stan.smith at intel.com] 
> Sent: Wednesday, November 07, 2007 9:47 PM
> To: Diego Guella; Tzachi Dar; Sufficool, Stanley; Hefty, Sean
> Cc: ofw at lists.openfabrics.org
> Subject: RE: [ofw] some WinOF questions
> 
> > Am I wrong? Is there some way I can use SDP in WinOF? How can I do 
> > that?
> 
> Microsoft requires those who distribute SDP to sign a license 
> with MS concerning IP rights of SDP.
> Since WinOF is open-source MS has declined to offer an SDP 
> licensing option, therefore SDP is not part of WinOF (Windows 
> OpenFabrics which is based on the openib-windows src tree).

[I'm an Engineer not a lawyer, so here is how I understand this, I might
be very wrong about this]
Microsoft indeed requires a license to distribute SDP in source code.
As a result it can not be an open source program. Still, one can use
binaries that it receives from Mellanox in order to use SDP with the
open fabrics software.
In order to get the source code of SDP in order to be able to contribute
one has to sign an agreement with MS. This is indeed a barrier, but it
can be passed, and was already passed twice.
So, on the practical reason, one can use SDP (binaries) with the open
fabrics code.

[again, I'm an Engineer not a lawyer, so here is how I understand this,
I might be very wrong about this]

> On the other hand, Mellanox has signed a licensing agreement 
> with MS and therefore distributes SDP as part of their WinIB 
> stack which is also based on the openIB-windows src plus 
> Mellanox value add.
> 
> The WinOF answer to missing SDP is WSD (Winsock Direct 
> Provider) which from the user's perspective provides similar 
> utilization of InfiniBand hardware as does SDP.
Please note that WSD has many drawbacks, which makes it practically
useless:
1) No support for XP or Vista. 
2) No interoperability with Linux or any other OS.

Both 1,2 are MS design that doesn't seem to change soon. This removes a
big part of the market.
3) Very low performance (compared for GE for example):
	a - BW for small messages is around GE. Only for messages
          that are bigger than 500K we start to see the real advantages
of it.
	b - High latency, almost double the latency of SDP. (~12us on
WSD compared to 7us on SDP).
	c - Connection rate of around 100 CPS compared to more than 2000
CPS on GE.

4) No support for UDP or Multicast.
	Looking at the spec, SDP doesn't support multicast or 
      UDP as well, but they can be added relatively easily.

> Within the Linux world, connected mode IPoIB approaches or in 
> some cases surpasses SDP performance.
> Connected mode IPoIB is appearing on the WinOF 2008 horizon.
A few words about connected mode on windows. Mellanox has made an
experiment to use ipoib CM on windows in the past. Results showed that
this can double the BW compared to IPOIB (UD). Although this seems a
step forward, it still means that using a single connection, one will
not be able to reach more than 600 MB/sec, and even that comes when
messages were around 256KB.
On SDP one can reach more than 1300 MB with messages as small as 8kb.
More than that, I claim that if we will work on SDP, we will be able to
reach wire speed with messages as small as 1.5 kb. This numbers are
better than what one can dream on using IPOIB CM on Linux and windows
together.
Please note that when implemented correctly, SDP has to offer 3
important things that IPOIB cm can't:
1) OS by pass.
2) Reliability in HW.
3) Zero copy.

Results that were achieved on Linux didn't really use 1 or 3, and
therefore still have a lot of place to improve.
[Please note that the experiment was very limited and performance in the
release mode might change]
 
> Stan.
> 
> 
> Diego Guella wrote:
> > Sean, Stanley, Tzachi, thanks for your answers.
> > 
> > I had downloaded WinIB 1.3 from Mellanox website, and 
> installed on my 
> > Dell Dimension 9100 with a MHES14-XTC and Windows XP.
> > I powered up my Dell PowerEdge 2850 with a MHGA28-XTC, SUSE 9.3 and 
> > OFED 1.1 (including OpenSM).
> > The two systems are connected back-to-back each other.
> > 
> > After configuring the IP addresses, IPoIB works fine (tried ping, 
> > VNC).
> > 
> > 
> > I then uninstalled WinIB and installed WinOF 1.0: same situation, 
> > IPoIB works fine as before.
> > However, in WinIB 1.3 installation there were two options that are 
> > missing in WinOF 1.0: for installing SDP and for install SDK.
> > 
> > 
> > 
> > My next step is to write a simple benchmark program that sends some 
> > data in both directions.
> > I would try it using IPoIB and SDP.
> > In WinIB 1.3 there were a "SDP" directory in the WinIB installation 
> > dir, a README instructing how to make a program use SDP.
> > These are missing in WinOF, so it seems SDP is not present.
> > 
> > 
> > Am I wrong? Is there some way I can use SDP in WinOF? How can I do 
> > that?
> > 
> > 
> > 
> > Thanks,
> > Diego
> > 
> > _______________________________________________
> > ofw mailing list
> > ofw at lists.openfabrics.org
> > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
>