[ofw] CM ref counting issues...

[Sean Hefty]

>This is showing that DREQs were sent for CEPs.  The CEPs were 'destroyed', but
>the DREQs were not canceled.  So, destruction of the CEPs hung until the DREQs
>timed out.

I was finally able to track down what appears to be the root of the problems.
The __cep_mad_send_cb() handling of failed requests is incorrect.  The callback
assumes that the mad being processed is associated with the current state of the
CEP, which may not be the case.

For example, in my testing, a REP mad was completed as canceled; however, by the
time the callback was invoked, the connection had already transitioned to the
DREQ_SENT state.  The result was that the state machine transitioned into
timewait, leaving the DREQ outstanding.

The implementation of the state machine in the CM code is subtle, but I think
the following untested change could fix the issue in all cases:

	KeAcquireInStackQueuedSpinLockAtDpcLevel( &gp_cep_mgr->lock, &hdl );
	/* Clear the sent MAD pointer so that we don't try cancelling again. */
	if( p_cep->p_send_mad == p_mad )
		p_cep->p_send_mad = NULL;
+	else
+	{
+		KeReleaseInStackQueuedSpinLockFromDpcLevel( &hdl );
+		ib_put_mad( p_mad );
+	}

If this works, it seems like the simplest fix.  The linux CM handled this case
by recording the state that a mad was sent in, then checked that the cep was in
the same state when it completed.  If the states differed, then the mad was
discarded.  This is another possibility if the above doesn't work.

- Sean