[Openib-windows] WSD: Behavior when the other side has not yet called accept, is different from TCP/IP (Ethernet)

[Tzachi Dar]

See bellow:

Thanks
Tzachi 

> -----Original Message-----
> From: ftillier.sst at gmail.com [mailto:ftillier.sst at gmail.com] 
> On Behalf Of Fabian Tillier
> Sent: Thursday, August 10, 2006 12:46 AM
> To: Tzachi Dar
> Cc: openib-windows at openib.org
> Subject: Re: [Openib-windows] WSD: Behavior when the other 
> side has not yet called accept, is different from TCP/IP (Ethernet)
> 
> Hi Tzachi,
> 
> On 8/9/06, Tzachi Dar <tzachid at mellanox.co.il> wrote:
> > Hi Fab,
> >
> > I have tested your fix with the following scenarios:
> >
> > 1) Client listening  -Connecting succeeds -
> > 2) No client listening on the remote side. Works fine - there is a 
> > fall back to ipoib and the right error is returned.
> >
> > 3) Many clients connects to a server. This was somewhat 
> problematic: I 
> > was using iperf 2.01 to test that and got some strange results: By 
> > default Iperf calls listen with a backlog of 5. When more than 5 
> > threads try to connect from the client side the server might return 
> > the new error, and this causes iperf to crash.
> 
> Does the client side of iperf crashes, or the server side?  
> The server side certainly shouldn't be crashing unless I 
> screwed up the patch...
> 
The client side is crashing, the server seems to be fine.

> The client side's connection request might time out, in which 
> case it would get IB_REJ_TIMEOUT which gets translated to 
> WSAETIMEOUT.  The WSD switch should then cause this to retry 
> via IPoIB on the client side.
> 

Maybe I got something here wrong in my system, but looking at network
traffic, there is sometimes a retry through IPOIB and sometimes there
isn't. I can also see from the logs that the rejection reason is 28
(IB_REJ_USER_DEFINED) (message is "[2720] socket 0009CFB0 connect
reject, reason=28") which is not what I expected to see. 


> > The immediate reason for the
> > crash is defiantly in Iperf (division by zero). I didn't 
> have the time 
> > to debug Iperf to find out what the real problem was.
> > Still, the fix that I was thinking about was in the function 
> > cm_req_callback to simply drop the packets, and therefore forcing a 
> > retry.
> 
> We can't just ignore the cm_req_callback - the REQ was 
> received, and the kernel CM created a connection end point 
> for it that it is tracking.  We won't get another callback 
> for that connection end point.  We'd have to create a queue 
> of requests beyond the backlog, which seems like the wrong 
> thing to do.  In any case, the end result shouldn't be much 
> different than what I implemented.  The patch I sent will 
> tell the CM to destroy it's connection end point, but it 
> won't send a REJ.  This results in the REQ on the remote side 
> being retried, and when received a new connection end point 
> is created and the REQ callback is invoked.
> 
> > What do you think? Do you have a better program to try, or 
> do we want 
> > to trust Iperf results?
> 
> Can netperf be used to test this scenario?  I'm not that 
> familiar with either of them to tell you the truth.
> 
Netperf doesn't work because it has another problem with WSD. In short
the problem is that Duplicate socket fails because the call to
openprocess fails with error access denied (at least this is what I
remember).


> In any case, do you agree about the CEP leak in the passive 
> side reject case?  
Yes.
>Can I check this in?  I'd like to check it 
> all in because it *seems* like the right thing to do to me, 
> but I don't want to introduce too much thrash in WSD at this 
> point without getting some buy-in.
> 
You can check this in (this is one step forward), but I'm not sure that
this is the right thing to do. After all the problem is not that
important, and we would like to get things stabilized for WHQL. We
probably need one more day to work on this to understand where the
problem is (maybe just in Iperf), and at least speaking for my self I
don't fill that I have the time for that right now. 



> Thanks,
> 
> - Fab
>