[ofw] RE: [PATCH] NDProvider: Use private heap for memory allocations

[Fab Tillier]

>> Which heap you use is up to you.  The SecureMemoryCacheCallback
>> requires that
> 
> The lower level libraries aren't calling SecureMemoryCacheCallback. The
> apps (upper libraries) are.

Apps and libraries never call SecureMemoryCacheCallback, that's just a prototype for the callback that they register with the OS.  The OS calls the callbacks when necessary.

> Why can't the apps use a different heap for this?

Apps don't know that memory is being registered.  Middleware libraries are registering the memory on behalf of the app, and caching the registrations so that an app that reuses its buffers don't pay the registration penalty for every I/O operation.

The kernel drivers, when you register memory, are calling MmSecureVirtualMemory.  This mechanism is what allows the SecureMemoryCacheCallback functionality to work - when memory gets freed that is secured, the OS invokes all the cache callbacks registered for the process freeing the memory.  Once the cache callbacks have been invoked, it attempts to free the memory again - the memory must be unsecured from the cache callback for it to get released.

So the cache callback deregisters the memory, which in the kernel calls MmUnsecureVirtualMemory, and then the memory succeeds in being freed.  

> Look at it this way.  The ND provider, winverbs, libibverbs, etc aren't
> doing anything wrong, yet they're having to change because something
> breaks in higher level code.

The kernel driver is what secures the memory.  If you want to be able to use the SecureMemoryCacheCallback functionality provided by the OS, you must abide by the rules for the callback: you can't touch the heap for which memory is being freed.  If you don't want to play by those OS rules, you can't cache memory registrations.

So it comes down to whether having a registration cache is something beneficial, especially if the OS assists in flushing stale entries.  MSMPI has a cache.  The WSD provider has a cache.  Those aren't going away, because there are real performance benefits in having them.  However, the drivers (the ones that are securing the memory when it is registered) aren't doing the right thing for the cache to work.  It is a driver-level problem, not an app problem.

-Fab