[ofw] IpoIB NDIS6 BSOD crash while loading driver

Alex Naslednikov xalex at mellanox.co.il
Sun Nov 1 01:26:17 PDT 2009 

Hello,
We will update the trunk soon and this problem should be already solved

XaleX 

-----Original Message-----
From: ofw-bounces at lists.openfabrics.org
[mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Smith, Stan
Sent: Friday, October 30, 2009 12:32 AM
To: Tzachi Dar; Leonid Keller
Cc: ofw at lists.openfabrics.org
Subject: [ofw] IpoIB NDIS6 BSOD crash while loading driver

Hello,
  The following crash occurs every time the IPoiB_NDIS6_CN driver is
loaded on a mthca HCA system.
Sometimes this crash occasionally with a ConnectX HCA?
In reviewing the text I realized I was using an x86 windbg with an x64
target. Will reinstall x64 windbg with a checked version of IPoIB
driver.
In the meantime, I thought you might be able to pin-point why s_buf == 0
?

Stan.


1: kd> !analyze -v
************************************************************************
*******
*
*
*                        Bugcheck Analysis
*
*
*
************************************************************************
*******

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address
at an interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 0000000000000008, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000000, bitfield :
        bit 0 : value 0 = read operation, 1 = write operation
        bit 3 : value 0 = not an execute operation, 1 = execute
operation (only on chips which support this level of status)
Arg4: fffff80001f3762c, address which referenced memory

Debugging Details:
------------------

*** ERROR: Symbol file could not be found.  Defaulted to export symbols
for hal.dll -
*** ERROR: Symbol file could not be found.  Defaulted to export symbols
for NDIS.SYS -
*** ERROR: Module load completed but symbols could not be loaded for
pacer.sys
*** ERROR: Symbol file could not be found.  Defaulted to export symbols
for tcpip.sys -
*** ERROR: Module load completed but symbols could not be loaded for
afd.sys
***** Kernel symbols are WRONG. Please fix symbols to do analysis.

*** ERROR: Module load completed but symbols could not be loaded for
mssmbios.sys
************************************************************************
*
***
***
***
***
***    Your debugger is not using the correct symbols
***
***
***
***    In order for this command to work properly, your symbol path
***
***    must point to .pdb files that have full type information.
***
***
***
***    Certain .pdb files (such as the public OS symbols) do not
***
***    contain the required information.  Contact the group that
***
***    provided you with these symbols if you need this command to
***
***    work.
***
***
***
***    Type referenced: nt!_KPRCB
***
***
***
************************************************************************
*

MODULE_NAME: ipoib

FAULTING_MODULE: fffff80001a1b000 nt

DEBUG_FLR_IMAGE_TIMESTAMP:  4ae8caf8

READ_ADDRESS: unable to get nt!MmSpecialPoolStart unable to get
nt!MmSpecialPoolEnd unable to get nt!MmPoolCodeStart unable to get
nt!MmPoolCodeEnd
 0000000000000008

CURRENT_IRQL:  0

FAULTING_IP:
hal!HalFreeCommonBuffer+490
fffff800`01f3762c 488b5a08        mov     rbx,qword ptr [rdx+8]

DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT

BUGCHECK_STR:  0xA

LAST_CONTROL_TRANSFER:  from fffff80001b1f502 to fffff80001a6f6f0

STACK_TEXT:
fffffa60`077b97c8 fffff800`01b1f502 : fffffa80`056597c0
00000000`00000065 00000000`00000008 fffff800`01ab2210 :
nt!DbgBreakPointWithStatus fffffa60`077b97d0 fffff800`01b202bb :
00000000`00000003 00000000`00000000 fffff800`01aafaa0 00000000`0000000a
: nt!HeadlessDispatch+0x192 fffffa60`077b9830 fffff800`01a75554 :
00000000`00000000 00000000`00000000 00000000`00000000 00000008`00000000
: nt!KeEnterKernelDebugger+0x73b fffffa60`077b9ea0 fffff800`01a751ee :
00000000`0000000a 00000000`00000008 00000000`00000002 00000000`00000000
: nt!KeBugCheckEx+0x104 fffffa60`077b9ee0 fffff800`01a740cb :
00000000`00000000 00000000`00000000 fffffa80`050eadf4 fffffa80`050eaca0
: nt!ZwUnloadKeyEx+0x23ce fffffa60`077ba020 fffff800`01f3762c :
00000000`00000050 fffffa80`050eade6 fffffa80`05863000 fffffa60`077bad00
: nt!ZwUnloadKeyEx+0x12ab fffffa60`077ba1b0 fffffa60`00809c91 :
fffffa80`050eaca0 00000000`c0000001 fffffa80`050eaba0 00000000`00000000
: hal!HalFreeCommonBuffer+0x490 fffffa60`077ba210 fffffa60`087a055b :
fffffa60`087ab490 fffffa80`050eaba0 fffffa80`05861f20 fffffa80`078fc708
: NDIS!NdisMFreeNetBufferSGList+0x31
fffffa60`077ba250 fffffa60`0879e168 : fffffa80`05861f20
fffffa80`050ead80 fffffa80`050eaba0 00000000`00000050 :
ipoib!__send_complete_net_buffer+0x7b
[f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib
_port.cpp @ 5489] fffffa60`077ba290 fffffa60`0879e4c3 :
fffffa80`05666108 00000000`00000000 fffffa80`078fc708 fffffa80`05861f20
: ipoib!ipoib_process_sg_list_real+0x5d0
[f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib
_port.cpp @ 3928] fffffa60`077ba390 fffff800`01f374ef :
00000000`00000000 fffffa80`078fc708 00000000`00000050 00000000`00000000
: ipoib!ipoib_process_sg_list+0x67
[f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib
_port.cpp @ 3987] fffffa60`077ba3c0 fffffa60`00809889 :
fffffa80`050eaca0 fffffa80`050eadc0 00000000`00000011 00000000`00000000
: hal!HalFreeCommonBuffer+0x353 fffffa60`077ba430 fffffa60`087a041c :
fffffa80`0667c1a0 fffffa60`077ba7d0 fffffa80`05861f20 fffffa80`050eaba0
: NDIS!NdisMAllocateNetBufferSGList+0x179
fffffa60`077ba4d0 fffffa60`0879353a : 00000000`00000000
00000000`00000000 00000000`00000001 00000000`00000000 :
ipoib!ipoib_port_send+0x34c
[f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib
_port.cpp @ 5439] fffffa60`077ba540 fffffa60`0080a122 :
fffffa60`08793440 00000000`00000000 fffffa80`050eaba0 fffffa80`0667c1a0
: ipoib!ipoib_send_net_buffer_list+0xfa
[f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib
_driver.cpp @ 3334] fffffa60`077ba580 fffffa60`06fafea7 :
00000000`00000000 00000000`00000000 fffffa60`00860110 fffffa80`0667c1a0
: NDIS!NdisFSendNetBufferLists+0xe2 fffffa60`077ba5d0 fffffa60`0080a1ae
: fffffa80`050eaba0 fffffa80`050eaba0 fffffa80`00000000
00000000`00000000 : pacer+0x6ea7 fffffa60`077ba6d0 fffffa60`009ac189 :
ffff0000`078fa5ac fffffa80`0667c1a0 00000000`00000000 00000000`00000000
: NDIS!NdisFSendNetBufferLists+0x16e
fffffa60`077ba710 fffffa60`00e6bcc4 : fffffa80`06ccd010
00000000`0000000e fffffa60`077ba7e0 fffffa60`077bac00 :
NDIS!NdisSendNetBufferLists+0x69 fffffa60`077ba760 fffffa60`00e6f3f8 :
00000000`00000000 00000000`00000800 00000000`00000001 00000000`00000000
: tcpip+0x62cc4 fffffa60`077ba860 fffffa60`00e6e85a : 00000000`00000000
fffffa80`00000011 fffffa80`00000014 00000000`00000000 : tcpip+0x663f8
fffffa60`077ba990 fffffa60`00e5632e : 00000000`00000000
fffffa60`00a04d0c 00000000`00000050 fffffa80`03aeaf00 : tcpip+0x6585a
fffffa60`077badb0 fffffa60`00e3c5da : 00000000`00000000
fffffa80`03aeaf00 fffffa80`04ccc350 fffffa60`077bb450 : tcpip+0x4d32e
fffffa60`077badf0 fffffa60`00e3c7f5 : fffffa60`077bb500
fffffa60`077bb570 fffffa60`077b0006 fffffa60`077b4300 : tcpip+0x335da
fffffa60`077bb420 fffff800`01a86de2 : 00000000`00000007
00000000`00000000 00000000`00000000 00000000`00000001 : tcpip+0x337f5
fffffa60`077bb450 fffffa60`00e3c856 : fffffa60`077bb570
fffffa80`050eb7e0 fffffa80`067128d0 fffffa60`077bb840 :
nt!KeExpandKernelStackAndCalloutEx+0xb2
fffffa60`077bb4d0 fffffa60`06f35b6a : fffffa60`005ee180
fffffa80`06b9e798 fffffa80`05639340 fffff800`01a8aa1e : tcpip+0x33856
fffffa60`077bb530 fffffa60`06f358bb : fffffa80`050eb7e0
fffffa60`077bbca0 fffffa80`05639340 fffffa60`077bb840 : afd+0x3bb6a
fffffa60`077bb5c0 fffffa60`06f1dc9f : fffffa80`05639340
fffffa60`077bb840 00000000`0000012c 00000000`00000000 : afd+0x3b8bb
fffffa60`077bb6b0 fffff800`01cd7aec : 00000000`00000000
fffffa60`06f1d010 fffffa80`06ca5270 fffffa80`06b62a01 : afd+0x23c9f
fffffa60`077bba10 fffff800`01ce07c6 : 00000000`00000001
00000000`00000524 00000000`00000000 00000000`00000000 :
nt!NtQueryInformationThread+0x208c
fffffa60`077bbb40 fffff800`01a74ef3 : 00000000`00000000
fffff800`01cd563f 00000000`00000000 00000000`00000006 :
nt!NtDeviceIoControlFile+0x56 fffffa60`077bbbb0 00000000`774b6d8a :
000007fe`fc5979c7 00000000`00000000 00000000`00000000 00000000`00000000
: nt!ZwUnloadKeyEx+0x20d3
00000000`01e6e058 000007fe`fc5979c7 : 00000000`00000000
00000000`00000000 00000000`00000000 00000000`00000000 :
ntdll!RtlpLocateActivationContextSection+0x185
00000000`01e6e060 00000000`00000000 : 00000000`00000000
00000000`00000000 00000000`00000000 00000000`01e6e168 :
mswsock!ReleaseWsaBufArray+0x31


STACK_COMMAND:  kb

FOLLOWUP_IP:
ipoib!__send_complete_net_buffer+7b
[f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib
_port.cpp @ 5489]
fffffa60`087a055b 488b7330        mov     rsi,qword ptr [rbx+30h]

FAULTING_SOURCE_CODE:
  5485:
s_buf->p_port->p_adapter->NdisMiniportDmaHandle,
  5486:
s_buf->p_sgl,
  5487:
s_buf->p_curr_nb);
  5488:
> 5489:         if( s_buf->p_send_buf )
  5490:         {
  5491:                 cl_perf_start( FreeSendBuf );
  5492:                 NdisFreeToNPagedLookasideList(
&s_buf->p_port->buf_mgr.send_buf_list,
  5493:                         s_buf->p_send_buf );
  5494:                 cl_perf_stop( &p_port->p_adapter->perf,
FreeSendBuf );


SYMBOL_STACK_INDEX:  8

SYMBOL_NAME:  ipoib!__send_complete_net_buffer+7b

FOLLOWUP_NAME:  MachineOwner

IMAGE_NAME:  ipoib.sys

BUCKET_ID:  WRONG_SYMBOLS

Followup: MachineOwner
---------

nt!DbgBreakPointWithStatus
nt!HeadlessDispatch+0x192
nt!KeEnterKernelDebugger+0x73b
nt!KeBugCheckEx+0x104
nt!ZwUnloadKeyEx+0x23ce
nt!ZwUnloadKeyEx+0x12ab (TrapFrame @ fffffa60`077ba020)
hal!HalFreeCommonBuffer+0x490
NDIS!NdisMFreeNetBufferSGList+0x31
ipoib!__send_complete_net_buffer(struct ipoib_send_NB_SG_t * s_buf =
0xfffffa80`05861f20, int status = 84848000, unsigned long compl_flags =
0x50eaba0, int bLock = 80)+0x7b
[f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib
_port.cpp @ 5489] ipoib!ipoib_process_sg_list_real(struct _DEVICE_OBJECT
* pDO = 0xfffffa80`05666108, void * pIrp = 0x00000000`00000000, struct
_SCATTER_GATHER_LIST * p_sgl = 0xfffffa80`078fc708, void * context =
0xfffffa80`05861f20)+0x5d0
[f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib
_port.cpp @ 3928] ipoib!ipoib_process_sg_list(struct _DEVICE_OBJECT *
pDO = 0x00000000`00000000, void * pIrp = 0xfffffa80`078fc708, struct
_SCATTER_GATHER_LIST * p_sgl = 0x00000000`00000050, void * context =
0x00000000`00000000)+0x67
[f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib
_port.cpp @ 3987]
hal!HalFreeCommonBuffer+0x353
NDIS!NdisMAllocateNetBufferSGList+0x179
ipoib!ipoib_port_send(struct _ipoib_port * p_port = 0x00000000`00000000,
struct _NET_BUFFER_LIST * p_net_buffer_list = 0x00000000`00000000,
unsigned long send_flags = 1)+0x34c
[f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib
_port.cpp @ 5439] ipoib!ipoib_send_net_buffer_list(void *
adapter_context = 0xfffffa60`08793440, struct _NET_BUFFER_LIST *
net_buffer_list = 0x00000000`00000000, unsigned long port_num =
0x50eaba0, unsigned long send_flags = 0x667c1a0)+0xfa
[f:\openib-windows-svn\latest\gen1\trunk\ulp\ipoib_ndis6_cm\kernel\ipoib
_driver.cpp @ 3334]
NDIS!NdisFSendNetBufferLists+0xe2
pacer+0x6ea7
NDIS!NdisFSendNetBufferLists+0x16e
NDIS!NdisSendNetBufferLists+0x69
tcpip+0x62cc4
tcpip+0x663f8
tcpip+0x6585a
tcpip+0x4d32e
tcpip+0x335da
tcpip+0x337f5
nt!KeExpandKernelStackAndCalloutEx+0xb2
tcpip+0x33856
afd+0x3bb6a
afd+0x3b8bb
afd+0x23c9f
_______________________________________________
ofw mailing list
ofw at lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw

More information about the ofw mailing list