[ofw] RE: [HW] memory allocation improvement in user space

[Fab Tillier]

Hi Leo,

>From: Leonid Keller
>Sent: Monday, June 23, 2008 6:44 AM
>
>Investigation of INSUFFICIENT_MEMORY failures on our stress tests
>brought us to "revelation", that VirtualAlloc function, used for
>implementation of posix_memalign, is a very "greedy" one: it allocates
>at least 64KB memory.
>As far as we usually ask one page, it is 16 times more than necessary.

VirtualAlloc doesn't actually allocate 64K of memory, but it does reserve 64K of virtual address space.  It then commits a single page.  This article: http://msdn.microsoft.com/en-us/library/ms810627.aspx explains this behavior: "The minimum size that can be reserved is 64K ... Requesting one page of reserved addresses results in a 64K address range" in the section about reserving memory.

>Presented below a patch, which implements posix_memalign with
>(ultimately) HeapAlloc functions.
>The patch was tested and worked OK.
>
>An important nuance, that was revealed during testing is as follows:
>A system function, which releases the resources of an exiting process,
>damages in some way the work of MmSecureVirtualMemory function, which we
>use today to secure CQ\QP\SRQ circular buffers and user buffers.
>If an application gets killed or exits without releasing the resources,
>IBBUS catches this event, starts its cascading destroy of resources and
>crashes on MmUnsecureVirtualMemory.
>Putting MmUnsecureVirtualMemory in try-except block saves from the
>crash, but an async thread, releasing QPs, freezes on
>MmUnsecureVirtualMemory, which fails to get some mutex.

Maybe the first exception didn't release the mutex?

>As far as there is no real reason to secure circular  buffers, i've
>solved the problem by skipping securing for IB objects.
>User buffers are still secured while memory registration.
>
>Therefore the patch contains 3 kinds of changes:
>1) new implementation of posix_memalign and all related to that;
>2) try-except block around MmUnsecureVirtualMemory;

I'd be weary of just working around the issue without understanding why it crashes, especially since you found some negative ramifications to this (the subsequent hang).  What if there's some memory corruption problem in the UVP - shouldn't that be fixed rather than hidden?

>3) new parameter in ib_umem_get and mthca_reg_virt_mr for skipping
>memory securing and all related to it;

-Fab