[ofw] IpoIB NDIS6 BSOD crash while loading driver

Thu Oct 29 15:32:12 PDT 2009

Hello,
  The following crash occurs every time the IPoiB_NDIS6_CN driver is loaded on a mthca HCA system.
Sometimes this crash occasionally with a ConnectX HCA?
In reviewing the text I realized I was using an x86 windbg with an x64 target. Will reinstall x64 windbg with a checked version of IPoIB driver.
In the meantime, I thought you might be able to pin-point why s_buf == 0 ?

Stan.

1: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 0000000000000008, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000000, bitfield :
        bit 0 : value 0 = read operation, 1 = write operation
        bit 3 : value 0 = not an execute operation, 1 = execute operation (only on chips which support this level of status)
Arg4: fffff80001f3762c, address which referenced memory

Debugging Details:
------------------

*** ERROR: Symbol file could not be found.  Defaulted to export symbols for hal.dll -
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for NDIS.SYS -
*** ERROR: Module load completed but symbols could not be loaded for pacer.sys
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for tcpip.sys -
*** ERROR: Module load completed but symbols could not be loaded for afd.sys
***** Kernel symbols are WRONG. Please fix symbols to do analysis.

*** ERROR: Module load completed but symbols could not be loaded for mssmbios.sys
*************************************************************************
***                                                                   ***
***                                                                   ***
***    Your debugger is not using the correct symbols                 ***
***                                                                   ***
***    In order for this command to work properly, your symbol path   ***
***    must point to .pdb files that have full type information.      ***
***                                                                   ***
***    Certain .pdb files (such as the public OS symbols) do not      ***
***    contain the required information.  Contact the group that      ***
***    provided you with these symbols if you need this command to    ***
***    work.                                                          ***
***                                                                   ***
***    Type referenced: nt!_KPRCB                                     ***
***                                                                   ***
*************************************************************************

MODULE_NAME: ipoib

FAULTING_MODULE: fffff80001a1b000 nt

DEBUG_FLR_IMAGE_TIMESTAMP:  4ae8caf8

READ_ADDRESS: unable to get nt!MmSpecialPoolStart
unable to get nt!MmSpecialPoolEnd
unable to get nt!MmPoolCodeStart
unable to get nt!MmPoolCodeEnd
 0000000000000008

CURRENT_IRQL:  0

FAULTING_IP:
hal!HalFreeCommonBuffer+490
fffff800`01f3762c 488b5a08        mov     rbx,qword ptr [rdx+8]

DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT

BUGCHECK_STR:  0xA

LAST_CONTROL_TRANSFER:  from fffff80001b1f502 to fffff80001a6f6f0

STACK_TEXT:
fffffa60`077b97c8 fffff800`01b1f502 : fffffa80`056597c0 00000000`00000065 00000000`00000008 fffff800`01ab2210 : nt!DbgBreakPointWithStatus
fffffa60`077b97d0 fffff800`01b202bb : 00000000`00000003 00000000`00000000 fffff800`01aafaa0 00000000`0000000a : nt!HeadlessDispatch+0x192
fffffa60`077b9830 fffff800`01a75554 : 00000000`00000000 00000000`00000000 00000000`00000000 00000008`00000000 : nt!KeEnterKernelDebugger+0x73b
fffffa60`077b9ea0 fffff800`01a751ee : 00000000`0000000a 00000000`00000008 00000000`00000002 00000000`00000000 : nt!KeBugCheckEx+0x104
fffffa60`077b9ee0 fffff800`01a740cb : 00000000`00000000 00000000`00000000 fffffa80`050eadf4 fffffa80`050eaca0 : nt!ZwUnloadKeyEx+0x23ce
fffffa60`077ba020 fffff800`01f3762c : 00000000`00000050 fffffa80`050eade6 fffffa80`05863000 fffffa60`077bad00 : nt!ZwUnloadKeyEx+0x12ab
fffffa60`077ba1b0 fffffa60`00809c91 : fffffa80`050eaca0 00000000`c0000001 fffffa80`050eaba0 00000000`00000000 : hal!HalFreeCommonBuffer+0x490
fffffa60`077ba210 fffffa60`087a055b : fffffa60`087ab490 fffffa80`050eaba0 fffffa80`05861f20 fffffa80`078fc708 : NDIS!NdisMFreeNetBufferSGList+0x31
fffffa60`077ba250 fffffa60`0879e168 : fffffa80`05861f20 fffffa80`050ead80 fffffa80`050eaba0 00000000`00000050 : ipoib!__send_complete_net_buffer+0x7b [f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib_port.cpp @ 5489]
fffffa60`077ba290 fffffa60`0879e4c3 : fffffa80`05666108 00000000`00000000 fffffa80`078fc708 fffffa80`05861f20 : ipoib!ipoib_process_sg_list_real+0x5d0 [f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib_port.cpp @ 3928]
fffffa60`077ba390 fffff800`01f374ef : 00000000`00000000 fffffa80`078fc708 00000000`00000050 00000000`00000000 : ipoib!ipoib_process_sg_list+0x67 [f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib_port.cpp @ 3987]
fffffa60`077ba3c0 fffffa60`00809889 : fffffa80`050eaca0 fffffa80`050eadc0 00000000`00000011 00000000`00000000 : hal!HalFreeCommonBuffer+0x353
fffffa60`077ba430 fffffa60`087a041c : fffffa80`0667c1a0 fffffa60`077ba7d0 fffffa80`05861f20 fffffa80`050eaba0 : NDIS!NdisMAllocateNetBufferSGList+0x179
fffffa60`077ba4d0 fffffa60`0879353a : 00000000`00000000 00000000`00000000 00000000`00000001 00000000`00000000 : ipoib!ipoib_port_send+0x34c [f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib_port.cpp @ 5439]
fffffa60`077ba540 fffffa60`0080a122 : fffffa60`08793440 00000000`00000000 fffffa80`050eaba0 fffffa80`0667c1a0 : ipoib!ipoib_send_net_buffer_list+0xfa [f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib_driver.cpp @ 3334]
fffffa60`077ba580 fffffa60`06fafea7 : 00000000`00000000 00000000`00000000 fffffa60`00860110 fffffa80`0667c1a0 : NDIS!NdisFSendNetBufferLists+0xe2
fffffa60`077ba5d0 fffffa60`0080a1ae : fffffa80`050eaba0 fffffa80`050eaba0 fffffa80`00000000 00000000`00000000 : pacer+0x6ea7
fffffa60`077ba6d0 fffffa60`009ac189 : ffff0000`078fa5ac fffffa80`0667c1a0 00000000`00000000 00000000`00000000 : NDIS!NdisFSendNetBufferLists+0x16e
fffffa60`077ba710 fffffa60`00e6bcc4 : fffffa80`06ccd010 00000000`0000000e fffffa60`077ba7e0 fffffa60`077bac00 : NDIS!NdisSendNetBufferLists+0x69
fffffa60`077ba760 fffffa60`00e6f3f8 : 00000000`00000000 00000000`00000800 00000000`00000001 00000000`00000000 : tcpip+0x62cc4
fffffa60`077ba860 fffffa60`00e6e85a : 00000000`00000000 fffffa80`00000011 fffffa80`00000014 00000000`00000000 : tcpip+0x663f8
fffffa60`077ba990 fffffa60`00e5632e : 00000000`00000000 fffffa60`00a04d0c 00000000`00000050 fffffa80`03aeaf00 : tcpip+0x6585a
fffffa60`077badb0 fffffa60`00e3c5da : 00000000`00000000 fffffa80`03aeaf00 fffffa80`04ccc350 fffffa60`077bb450 : tcpip+0x4d32e
fffffa60`077badf0 fffffa60`00e3c7f5 : fffffa60`077bb500 fffffa60`077bb570 fffffa60`077b0006 fffffa60`077b4300 : tcpip+0x335da
fffffa60`077bb420 fffff800`01a86de2 : 00000000`00000007 00000000`00000000 00000000`00000000 00000000`00000001 : tcpip+0x337f5
fffffa60`077bb450 fffffa60`00e3c856 : fffffa60`077bb570 fffffa80`050eb7e0 fffffa80`067128d0 fffffa60`077bb840 : nt!KeExpandKernelStackAndCalloutEx+0xb2
fffffa60`077bb4d0 fffffa60`06f35b6a : fffffa60`005ee180 fffffa80`06b9e798 fffffa80`05639340 fffff800`01a8aa1e : tcpip+0x33856
fffffa60`077bb530 fffffa60`06f358bb : fffffa80`050eb7e0 fffffa60`077bbca0 fffffa80`05639340 fffffa60`077bb840 : afd+0x3bb6a
fffffa60`077bb5c0 fffffa60`06f1dc9f : fffffa80`05639340 fffffa60`077bb840 00000000`0000012c 00000000`00000000 : afd+0x3b8bb
fffffa60`077bb6b0 fffff800`01cd7aec : 00000000`00000000 fffffa60`06f1d010 fffffa80`06ca5270 fffffa80`06b62a01 : afd+0x23c9f
fffffa60`077bba10 fffff800`01ce07c6 : 00000000`00000001 00000000`00000524 00000000`00000000 00000000`00000000 : nt!NtQueryInformationThread+0x208c
fffffa60`077bbb40 fffff800`01a74ef3 : 00000000`00000000 fffff800`01cd563f 00000000`00000000 00000000`00000006 : nt!NtDeviceIoControlFile+0x56
fffffa60`077bbbb0 00000000`774b6d8a : 000007fe`fc5979c7 00000000`00000000 00000000`00000000 00000000`00000000 : nt!ZwUnloadKeyEx+0x20d3
00000000`01e6e058 000007fe`fc5979c7 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlpLocateActivationContextSection+0x185
00000000`01e6e060 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`01e6e168 : mswsock!ReleaseWsaBufArray+0x31

STACK_COMMAND:  kb

FOLLOWUP_IP:
ipoib!__send_complete_net_buffer+7b [f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib_port.cpp @ 5489]
fffffa60`087a055b 488b7330        mov     rsi,qword ptr [rbx+30h]

FAULTING_SOURCE_CODE:
  5485:                                                         s_buf->p_port->p_adapter->NdisMiniportDmaHandle,
  5486:                                                         s_buf->p_sgl,
  5487:                                                         s_buf->p_curr_nb);
  5488:
> 5489:         if( s_buf->p_send_buf )
  5490:         {
  5491:                 cl_perf_start( FreeSendBuf );
  5492:                 NdisFreeToNPagedLookasideList( &s_buf->p_port->buf_mgr.send_buf_list,
  5493:                         s_buf->p_send_buf );
  5494:                 cl_perf_stop( &p_port->p_adapter->perf, FreeSendBuf );

SYMBOL_STACK_INDEX:  8

SYMBOL_NAME:  ipoib!__send_complete_net_buffer+7b

FOLLOWUP_NAME:  MachineOwner

IMAGE_NAME:  ipoib.sys

BUCKET_ID:  WRONG_SYMBOLS

Followup: MachineOwner
---------

nt!DbgBreakPointWithStatus
nt!HeadlessDispatch+0x192
nt!KeEnterKernelDebugger+0x73b
nt!KeBugCheckEx+0x104
nt!ZwUnloadKeyEx+0x23ce
nt!ZwUnloadKeyEx+0x12ab (TrapFrame @ fffffa60`077ba020)
hal!HalFreeCommonBuffer+0x490
NDIS!NdisMFreeNetBufferSGList+0x31
ipoib!__send_complete_net_buffer(struct ipoib_send_NB_SG_t * s_buf = 0xfffffa80`05861f20, int status = 84848000, unsigned long compl_flags = 0x50eaba0, int bLock = 80)+0x7b [f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib_port.cpp @ 5489]
ipoib!ipoib_process_sg_list_real(struct _DEVICE_OBJECT * pDO = 0xfffffa80`05666108, void * pIrp = 0x00000000`00000000, struct _SCATTER_GATHER_LIST * p_sgl = 0xfffffa80`078fc708, void * context = 0xfffffa80`05861f20)+0x5d0 [f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib_port.cpp @ 3928]
ipoib!ipoib_process_sg_list(struct _DEVICE_OBJECT * pDO = 0x00000000`00000000, void * pIrp = 0xfffffa80`078fc708, struct _SCATTER_GATHER_LIST * p_sgl = 0x00000000`00000050, void * context = 0x00000000`00000000)+0x67 [f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib_port.cpp @ 3987]
hal!HalFreeCommonBuffer+0x353
NDIS!NdisMAllocateNetBufferSGList+0x179
ipoib!ipoib_port_send(struct _ipoib_port * p_port = 0x00000000`00000000, struct _NET_BUFFER_LIST * p_net_buffer_list = 0x00000000`00000000, unsigned long send_flags = 1)+0x34c [f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib_port.cpp @ 5439]
ipoib!ipoib_send_net_buffer_list(void * adapter_context = 0xfffffa60`08793440, struct _NET_BUFFER_LIST * net_buffer_list = 0x00000000`00000000, unsigned long port_num = 0x50eaba0, unsigned long send_flags = 0x667c1a0)+0xfa [f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib_driver.cpp @ 3334]
NDIS!NdisFSendNetBufferLists+0xe2
pacer+0x6ea7
NDIS!NdisFSendNetBufferLists+0x16e
NDIS!NdisSendNetBufferLists+0x69
tcpip+0x62cc4
tcpip+0x663f8
tcpip+0x6585a
tcpip+0x4d32e
tcpip+0x335da
tcpip+0x337f5
nt!KeExpandKernelStackAndCalloutEx+0xb2
tcpip+0x33856
afd+0x3bb6a
afd+0x3b8bb
afd+0x23c9f