[ofw] CM ref counting issues...

[Hefty, Sean]

Here are stack dumps that touch the IB stack when the application hangs.  The trace for the hanging process is the last one listed.  The application is exiting, causing winverbs to cleanup in the kernel.  I have cm_destroy_id call DbgBreakPoint after it times out.  Also, note that both the client and server are running on the same system.

Looking at the list, nothing obvious stands out that should cause a deadlock.  Is there anything special about how MADs are processed in the loopback case, or the handling of the CM timewait state?

  4.00005c  fffffa80036c2720 fffe752f Blocked    nt!KiSwapContext+0x7f
                                        nt!KiSwapThread+0x2fa
                                        nt!KeWaitForSingleObject+0x2da
                                        winverbs!WvProviderDisableRemove+0x4a
                                        winverbs!WvQpAcquire+0x2a
                                        winverbs!WvEpDisconnectQp+0x31
                                        winverbs!WvEpDisconnectHandler+0xcd
                                        winverbs!WorkQueueHandler+0x4b
                                        nt!IopProcessWorkItem+0x27
                                        nt!ExpWorkerThread+0x11a
                                        nt!PspSystemThreadStartup+0x57
                                        nt!KxStartSystemThread+0x16
   4.0000cc  fffffa8003b7f890 ffffff9b Blocked    nt!KiSwapContext+0x7f
                                        nt!KiSwapThread+0x2fa
                                        nt!KeWaitForSingleObject+0x2da
                                        ibbus!cl_event_wait_on+0x33
                                        ibbus!__cl_thread_pool_routine+0x2b
                                        ibbus!__thread_callback+0x28
                                        nt!PspSystemThreadStartup+0x57
                                        nt!KxStartSystemThread+0x16
   4.0000e0  fffffa8003b87040 ffffff4b Blocked    nt!KiSwapContext+0x7f
                                        nt!KiSwapThread+0x2fa
                                        nt!KeWaitForMultipleObjects+0x2d6
                                        mthca!eq_polling_thread+0x77
                                        nt!PspSystemThreadStartup+0x57
                                        nt!KxStartSystemThread+0x16
   4.0000e4  fffffa8004028040 ffff3baf Blocked    nt!KiSwapContext+0x7f
                                        nt!KiSwapThread+0x2fa
                                        nt!KeWaitForSingleObject+0x2da
                                        ibbus!cl_event_wait_on+0x33
                                        ibbus!__cl_thread_pool_routine+0x2b
                                        ibbus!__thread_callback+0x28
                                        nt!PspSystemThreadStartup+0x57
                                        nt!KxStartSystemThread+0x16
   4.0000e8  fffffa8004028bb0 fffe7791 Blocked    nt!KiSwapContext+0x7f
                                        nt!KiSwapThread+0x2fa
                                        nt!KeWaitForSingleObject+0x2da
                                        ibbus!cl_event_wait_on+0x33
                                        ibbus!__cl_thread_pool_routine+0x2b
                                        ibbus!__thread_callback+0x28
                                        nt!PspSystemThreadStartup+0x57
                                        nt!KxStartSystemThread+0x16
   4.0000ec  fffffa8004028720 fffe7442 Blocked    nt!KiSwapContext+0x7f
                                        nt!KiSwapThread+0x2fa
                                        nt!KeWaitForSingleObject+0x2da
                                        ibbus!cl_event_wait_on+0x33
                                        ibbus!__cl_thread_pool_routine+0x2b
                                        ibbus!__thread_callback+0x28
                                        nt!PspSystemThreadStartup+0x57
                                        nt!KxStartSystemThread+0x16
   4.0000f0  fffffa8004031040 ffffff4b Blocked    nt!KiSwapContext+0x7f
                                        nt!KiSwapThread+0x2fa
                                        nt!KeWaitForSingleObject+0x2da
                                        mthca!__cl_thread_pool_routine+0x31
                                        mthca!__thread_callback+0x28
                                        nt!PspSystemThreadStartup+0x57
                                        nt!KxStartSystemThread+0x16
   4.0000f8  fffffa80043eabb0 ffff3baf Blocked    nt!KiSwapContext+0x7f
                                        nt!KiSwapThread+0x2fa
                                        nt!KeWaitForSingleObject+0x2da
                                        ipoib!__cl_thread_pool_routine+0x31
                                        ipoib!__thread_callback+0x28
                                        nt!PspSystemThreadStartup+0x57
                                        nt!KxStartSystemThread+0x16
6dc.000458  fffffa80076f4bb0 fffe7329 RUNNING    nt!DbgBreakPoint
                                        ibbus!cm_destroy_id+0x6c
                                        winverbs!WvEpFree+0x74
                                        winverbs!WvProviderCleanup+0x76
                                        Wdf01000!FxPkgGeneral::OnCleanup+0x82
                                        Wdf01000!FxPkgGeneral::Dispatch+0x1ce
                                        Wdf01000!FxDevice::Dispatch+0xa9
                                        nt!IopCloseFile+0x184
                                        nt!ObpDecrementHandleCount+0xc0
                                        nt!ObpCloseHandleTableEntry+0xb7
                                        nt!ObpCloseHandleProcedure+0x30
                                        nt!ExSweepHandleTable+0x73
                                        nt!ObKillProcess+0x61
                                        nt!PspExitThread+0x4b8
                                        nt!PspTerminateThreadByPointer+0x4d
                                        nt!NtTerminateProcess+0xfa
                                        nt!KiSystemServiceCopyEnd+0x13
                                        ntdll!ZwTerminateProcess+0xa
                                        ntdll!RtlExitUserProcess+0x8b
                                        msvcrt!cinit+0x13b
                                        +0xff474be8