[openib-general] Re: [PATCH 3/3] iWARP CM - iWARP ConnectionManager

[Steve Wise]

On Wed, 2006-03-22 at 10:28 -0800, Sean Hefty wrote:
> >Once the the cm_id is connected, the provider must post a CLOSE event
> >when it is done with the cm_id.  That's the model.  The IWCM will not
> >free the cm_id until the CLOSE upcall happens.   Adding an explicit
> >alloc_context/dealloc_context in the provider will just push this logic
> >down into each provider.  IE:  The chelsio provider would block the
> >dealloc_context call until the LLP connection is fully shut down.
> 
> I just think that this approach is susceptible subtle race conditions that will
> be extremely difficult to debug.  And so far all of the patches submitted have
> had some sort of race.  I do not know if there's a race in the latest
> submission.  I'm just saying that the destruction is complex -- involving a
> cm_id state, bit-flag, event state, and reference count -- which makes it
> difficult to verify its correctness.

We're struggling with implementing an approach that meets your
requirements -and- supports RDMAC verbs and iWARP providers...

> For example, as soon as the user calls connect(), can they receive a CLOSE
> event, even before the connect() call returns?  

No.  connect results in a CONNECT_REPLY event always. Not a CLOSE
event.   

Tom, can you post more info on the various events and their relationship
to the cm_id states?  Maybe that will help?

> If so, are there any issues
> here?  Is it possible for the user to call down to the provider, after the
> provider has generated a CLOSE event, resulting in accessing the wrong
> connection, or crashing in the provider?

The IWCM prevents this, methinks, by failing any downcalls once the
cm_id is no longer in a CONNECTED state.

> 
> Note, that I'm not saying that providers need to block a call until everything
> is shutdown.  It only needs to ensure that no callbacks will occur after
> dealloc_context() returns.  Destroy_listen() should be providing similar logic,
> so it ends up being in each provider anyway.

And destroy_listen() blocks until the RNIC acknowleges the destruction
of the listening endpoint.  

For orderly close, the provider needs to wait until the close completes
or is aborted due to protocol errors.  This can be implemented in many
ways, but blocking the caller is perhaps the simplest.  Also, note, that
the QP needs to stay around until it transitions out of CLOSING by the
provider.  This further complicates things and is different from IB.    


> At this point, I'm still trying to understand the operation.  When does the
> provider allocate a context for the user?  My guess is when calling connect() or
> listen().  When does the provider deallocate this context?  If it's not always
> done in response to the user invoking a function, then we're almost certain to
> have a race.

connect(), listen(), and when incoming iwarp connections get
established.

Deallocation for connected contexts can only happen after the TCP
protocol has finished tearing down the connection.  This could be via an
orderly close or an abortive close.

I hope I'm helping here...

Stevo.