[ofa-general] New proposal for memory management

[Barrett, Brian W]

On 4/30/09 16:01 , "Woodruff, Robert J" <robert.j.woodruff at intel.com> wrote:

> Brian wrote,
> 
>> There's an application at Sandia and at Los Alamos which both of which cause
>> problems for our linker tricks.  This leads to such things as (proven)
>> silent data corruption.  Perhaps your users are just getting silent data
>> corruption and not doing enough validation and verification to know it?  Or
>> maybe Intel's just gotten lucky - the majority of our applications seem to
>> have no issue with the registration cache.  But the outliers with proven
>> data corruption are the kind of things that keep me up at night.
> 
> Have you tried these applications with any MPI other than OpenMPI ?
> i.e., does this corruption happen with Intel MPI and other MPIs as well?
> If it is specific to OpenMPI, then perhaps it is just a bug in OpenMPI that
> can be fixed.

Of course it's a bug in Open MPI - it's one we can't fix (and is in MVAPICH
as well), but it's a bug in the MPI just the same.  I'm not in a position to
run such an app against Intel MPI, nor do I know what tricks you play to
compare it to MVAPICH and Open MPI.

>> We came with a real problem we're having with code development in real-world
>> applications, presented two solutions, and were essentially told to take a
>> hike.  If this sounds like a lot of whining to the OFA community, than the
>> OFA community shouldn't be surprised that the VERBS adoption rate is as poor
>> as it is.
> 
> Of the solutions that have been presented so far,
> I think the kernel notifier approach would be a better solution.
> 
> Besides the kernel bloat and complexity of a memory registration cache
> in the kernel, I am not sure it would really be able to work the
> way you would want. For example, the kernel has no way of knowing
> when some application calls free(), i.e., free() may not call the kernel
> to release the memory back to the kernel. It often just puts the free'd
> memory block on a free list within libc in user-space.
> 
> Thus, if we had a kernel memory registration cache, from the kernel's
> perspecive, this block of memory would appear to still be in use and
> could not be evicted from the cache. Thus the cache could end up filling
> up with lots of registrations for memory that has already been
> free()'d in user-space but are stitting on some free list in libc.

I don't actually care about the user calling free().  I care about events
that could change the virtual to physical mapping.  Intercepting free() is
actually really painful from a performance standpoint, as most apps
(especially C++ apps) call malloc/free orders of magnitude more times than
the app gives memory back to the kernel.

> This is another reason why I think the caching should be done in
> user-space. If the hooks do not exist in libc to hook
> all of the appropriate routines, then perhaps you should ask the
> libc maintainers to add what you need with perhaps the addition
> of the kernel notifier design that roland suggested.

Have you talked to the glibc malloc developers?  They're worse than this
group.  And since I have to intercept munmap and the SysV calls as well,
there are hooks in functions that really should be simple syscall wrappers,
which even in my opinion isn't defensible to add.

I'm done now.  You don't want to fix your crap, that's fine.  Just don't be
surprised by the continued "why you shouldn't use IB" presentations from
people who have to write applications to it.

Brian

--
   Brian W. Barrett
   Dept. 1423: Scalable System Software
   Sandia National Laboratories