<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=US-ASCII">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.45">
<TITLE>RE: Some more additions questions about the patch:</TITLE>
</HEAD>
<BODY>
<BR>
<BR>
<P><FONT SIZE=2>>-----Original Message-----</FONT>
<BR><FONT SIZE=2>>From: Fab Tillier [<A HREF="mailto:ftillier@silverstorm.com">mailto:ftillier@silverstorm.com</A>]</FONT>
<BR><FONT SIZE=2>>Sent: Monday, September 19, 2005 8:01 PM</FONT>
<BR><FONT SIZE=2>>To: 'Tzachi Dar'; openib-windows@openib.org</FONT>
<BR><FONT SIZE=2>>Subject: RE: Some more additions questions about the patch:</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>> From: Tzachi Dar [<A HREF="mailto:tzachid@mellanox.co.il">mailto:tzachid@mellanox.co.il</A>]</FONT>
<BR><FONT SIZE=2>>> Sent: Monday, September 19, 2005 12:58 AM</FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> >-----Original Message-----</FONT>
<BR><FONT SIZE=2>>> >From: Fab Tillier [<A HREF="mailto:ftillier@silverstorm.com">mailto:ftillier@silverstorm.com</A>]</FONT>
<BR><FONT SIZE=2>>> >Sent: Monday, September 19, 2005 7:51 AM</FONT>
<BR><FONT SIZE=2>>> >To: 'Tzachi Dar'; openib-windows@openib.org</FONT>
<BR><FONT SIZE=2>>> >Subject: RE: Some more additions questions about the patch:</FONT>
<BR><FONT SIZE=2>>> ></FONT>
<BR><FONT SIZE=2>>> >For the kernel, I'm considering going a step further an eliminating the</FONT>
<BR><FONT SIZE=2>>> >memory tracker - driver verifier has memory leak detection built in, and</FONT>
<BR><FONT SIZE=2>>> >we might as well just use that instead. The memory leak detection</FONT>
<BR><FONT SIZE=2>>> >provided by driver verifier doesn't require any sort of recompile of the</FONT>
<BR><FONT SIZE=2>>> >drivers, which is an extra benefit.</FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> Does the memory leak detection of the driver verifier also tell where the</FONT>
<BR><FONT SIZE=2>>leak</FONT>
<BR><FONT SIZE=2>>> was? If not than there is an advantage to our mechanism, if yes than our</FONT>
<BR><FONT SIZE=2>>> mechanism is not really needed.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>It does, but not immediately. So when the driver unloads, verifier will</FONT>
<BR><FONT SIZE=2>>issue a</FONT>
<BR><FONT SIZE=2>>bugcheck with bugcheck code 0xC4. The first parameter is a subcode which</FONT>
<BR><FONT SIZE=2>>tells</FONT>
<BR><FONT SIZE=2>>you the reason for the bugcheck. For reasons 0x60 and 0x62 tell you a</FONT>
<BR><FONT SIZE=2>>driver</FONT>
<BR><FONT SIZE=2>>was unloaded without freeing all its memory. The DDK docs describe how to</FONT>
<BR><FONT SIZE=2>>get</FONT>
<BR><FONT SIZE=2>>the detailed memory information. Here's what it looks like (when putting</FONT>
<BR><FONT SIZE=2>>in</FONT>
<BR><FONT SIZE=2>>place the patch to use the unload handler to call CL_DEINIG, I left in two</FONT>
<BR><FONT SIZE=2>>calls</FONT>
<BR><FONT SIZE=2>>to CL_INIT, so the memory tracker was never freed):</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>***************************************************************************</FONT>
<BR><FONT SIZE=2>>****</FONT>
<BR><FONT SIZE=2>>*</FONT>
<BR><FONT SIZE=2>>*</FONT>
<BR><FONT SIZE=2>>* Bugcheck Analysis</FONT>
<BR><FONT SIZE=2>>*</FONT>
<BR><FONT SIZE=2>>*</FONT>
<BR><FONT SIZE=2>>*</FONT>
<BR><FONT SIZE=2>>***************************************************************************</FONT>
<BR><FONT SIZE=2>>****</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Use !analyze -v to get detailed debugging information.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>BugCheck C4, {62, 8231de18, 8231ddc0, 20}</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Probably caused by : ipoib.sys</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Followup: MachineOwner</FONT>
<BR><FONT SIZE=2>>---------</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>nt!RtlpBreakWithStatusInstruction:</FONT>
<BR><FONT SIZE=2>>80871648 cc int 3</FONT>
<BR><FONT SIZE=2>>0: kd> dp ViBadDriver L1; dS @$p</FONT>
<BR><FONT SIZE=2>>808ab138 8231ddd0</FONT>
<BR><FONT SIZE=2>>8231de18 "ipoib.sys"</FONT>
<BR><FONT SIZE=2>>0: kd> !verifier 3 ipoib.sys</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Verify Level fb ... enabled options are:</FONT>
<BR><FONT SIZE=2>> special pool</FONT>
<BR><FONT SIZE=2>> special irql</FONT>
<BR><FONT SIZE=2>> all pool allocations checked on unload</FONT>
<BR><FONT SIZE=2>> Io subsystem checking enabled</FONT>
<BR><FONT SIZE=2>> Deadlock detection enabled</FONT>
<BR><FONT SIZE=2>> Enhanced Io checking enabled</FONT>
<BR><FONT SIZE=2>> DMA checking enabled</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Summary of All Verifier Statistics</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>RaiseIrqls 0x0</FONT>
<BR><FONT SIZE=2>>AcquireSpinLocks 0x955</FONT>
<BR><FONT SIZE=2>>Synch Executions 0x0</FONT>
<BR><FONT SIZE=2>>Trims 0x39</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Pool Allocations Attempted 0x34</FONT>
<BR><FONT SIZE=2>>Pool Allocations Succeeded 0x34</FONT>
<BR><FONT SIZE=2>>Pool Allocations Succeeded SpecialPool 0x34</FONT>
<BR><FONT SIZE=2>>Pool Allocations With NO TAG 0x0</FONT>
<BR><FONT SIZE=2>>Pool Allocations Failed 0x0</FONT>
<BR><FONT SIZE=2>>Resource Allocations Failed Deliberately 0x0</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Current paged pool allocations 0x0 for 00000000 bytes</FONT>
<BR><FONT SIZE=2>>Peak paged pool allocations 0x0 for 00000000 bytes</FONT>
<BR><FONT SIZE=2>>Current nonpaged pool allocations 0x20 for 00000E40 bytes</FONT>
<BR><FONT SIZE=2>>Peak nonpaged pool allocations 0x30 for 0044B300 bytes</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Driver Verification List</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Entry State NonPagedPool PagedPool Module</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>8231ddc0 Loaded 00000e40 00000000 ipoib.sys</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Current Pool Allocations 00000000 00000020</FONT>
<BR><FONT SIZE=2>>Current Pool Bytes 00000000 00000e40</FONT>
<BR><FONT SIZE=2>>Peak Pool Allocations 00000000 00000030</FONT>
<BR><FONT SIZE=2>>Peak Pool Bytes 00000000 0044b300</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>PoolAddress SizeInBytes Tag CallersAddress</FONT>
<BR><FONT SIZE=2>>82638f80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>82df4f80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>82612f80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>82e20f80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>82de2f80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>828f2f80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>82736f80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>82902f80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>827e4f80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>830e2f80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>82664f80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>82880f80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>8250af80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>82474f80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>82642f80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>831eef80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>831ecff8 0x00000008 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>831eaf80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>831e8fe0 0x00000020 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>831e6f80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>831e4fe0 0x00000020 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>831e2f80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>831e0fe0 0x00000020 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>831dcf80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>831dafe0 0x00000020 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>831d8f80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>831d6fa0 0x0000005c Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>831d4f80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>831d2ff8 0x00000008 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>831d0f80 0x00000080 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>831ceec0 0x0000013c Ddk b84ff776</FONT>
<BR><FONT SIZE=2>>831ccf68 0x00000098 Ddk b84ff776</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Using the address, you can show the disassembly using the 'u' command:</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>0: kd> u b84ff776</FONT>
<BR><FONT SIZE=2>>ipoib!__cl_malloc_priv+0x96</FONT>
<BR><FONT SIZE=2>>[c:\dev\openib\openib\base\core\complib\kernel\cl_memory_osd.c @ 51]:</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>This actually highlights an issue in complib's memory tracker - it obscures</FONT>
<BR><FONT SIZE=2>>the</FONT>
<BR><FONT SIZE=2>>actual caller by providing a wrapper to the system allocation calls, so all</FONT>
<BR><FONT SIZE=2>>allocations point to the internal complib allocator, rather than the</FONT>
<BR><FONT SIZE=2>>caller.</FONT>
<BR><FONT SIZE=2>OK, so I guess this can be safely removed.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>> >> 2) I have moved the place in which the adapter is being created.</FONT>
<BR><FONT SIZE=2>>It</FONT>
<BR><FONT SIZE=2>>> >> should better be created when the number of adapters in the list goes</FONT>
<BR><FONT SIZE=2>>> >> from 0 to 1, This is the case in which there is someone holding the</FONT>
<BR><FONT SIZE=2>>> >> device, the IPOIB ports are being disabled and re-enabled again.</FONT>
<BR><FONT SIZE=2>>> ></FONT>
<BR><FONT SIZE=2>>> >You mean when you register the device with NDIS for IOCTL access, or the</FONT>
<BR><FONT SIZE=2>>> >IPoIB adapter object? Can't we register the device from DriverEntry,</FONT>
<BR><FONT SIZE=2>>and</FONT>
<BR><FONT SIZE=2>>> >deregister it from the unload handler? The docs for the calls seem to</FONT>
<BR><FONT SIZE=2>>> >indicate that's the common usage model. I don't think the system will</FONT>
<BR><FONT SIZE=2>>> >invoke the unload handler until all file handles have been closed. This</FONT>
<BR><FONT SIZE=2>>> >would simplify things a great deal, wouldn't it?</FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> As far as what I saw, DriverUnload will not be called until we close the</FONT>
<BR><FONT SIZE=2>>> device, so although I agree that this is simpler, it doesn't work. (you</FONT>
<BR><FONT SIZE=2>>can</FONT>
<BR><FONT SIZE=2>>> play with it of course, maybe you will find something).</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Ok, I'll play with this and let you know if I can get it to work.</FONT>
<BR><FONT SIZE=2>Good luck.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>> >> 3) Please add the case of IRP_MJ_INTERNAL_DEVICE_CONTROL to the</FONT>
<BR><FONT SIZE=2>>> >> function __ipoib_dispatch. It should be handled just like</FONT>
<BR><FONT SIZE=2>>> >> IRP_MJ_DEVICE_CONTROL.</FONT>
<BR><FONT SIZE=2>>> ></FONT>
<BR><FONT SIZE=2>>> >Why don't kernel drivers use IRP_MJ_DEVICE_CONTROL? Why use</FONT>
<BR><FONT SIZE=2>>> >IRP_MJ_INTERNAL_DEVICE_CONTROL?</FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> I'm not sure why IRP_MJ_DEVICE_CONTROL doesn't work for drivers (I</FONT>
<BR><FONT SIZE=2>>believe</FONT>
<BR><FONT SIZE=2>>> that it should). I'm still trying to understand if I should use</FONT>
<BR><FONT SIZE=2>>> IoGetDeviceObjectPointer to open the device or ZwOpenFile. This might be</FONT>
<BR><FONT SIZE=2>>the</FONT>
<BR><FONT SIZE=2>>> cause of the problem (I'm currently using IoGetDeviceObjectPointer).</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>I think IoGetDeviceObjectPointer is the right one. Are you building the</FONT>
<BR><FONT SIZE=2>>IRP</FONT>
<BR><FONT SIZE=2>>yourself, or using IoBuildDeviceIoControlRequest? I don't think ZwOpenFile</FONT>
<BR><FONT SIZE=2>>will</FONT>
<BR><FONT SIZE=2>>work, as I haven't seen any docs to describe how to send IOCTL requests to</FONT>
<BR><FONT SIZE=2>>a</FONT>
<BR><FONT SIZE=2>>file in the kernel. I think you have to use IoCallDriver on a device</FONT>
<BR><FONT SIZE=2>>object,</FONT>
<BR><FONT SIZE=2>>which ZwOpenFile doesn't give you.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>I have made some more searched and I have found out an example in the DDK that says you should use ZwOpenFile.</FONT>
<BR><FONT SIZE=2>Look in the file WINDDK\3790.1830\src\network\ndis\ndiswdm\init.c</FONT>
<BR><FONT SIZE=2>It also shows how to get the device object in order to send IOCTLS.</FONT>
<BR><FONT SIZE=2>So it seems to me that this problem is over, although I still have to find how to close the device when IPOIB is trying to get down. I have some ideas and I'll try them tomorrow, if I have time.</FONT></P>
<P><FONT SIZE=2>>- Fab</FONT>
</P>
</BODY>
</HTML>