[openib-general] Question about pinning memory

[Roland Dreier]

    Pete> I did some thinking about this issue a while back and came
    Pete> up with a cooperative kernel/user implementation to track
    Pete> linux VM activity using existing vm_area_struct->vm_ops
    Pete> function hooks (i.e. no kernel patch required).  The MPI
    Pete> library essentially makes a system call before reusing a
    Pete> cached memory registration to verify it is still valid, and
    Pete> the kernel module keeps track of what happens to cached
    Pete> mappings as the VM system is exercised via sbrk, mmap, fork,
    Pete> etc.  It works for any sort of memory activity, including
    Pete> arbitrary mmap() of memory or files, since it plugs in at
    Pete> the basic VM interface level.

This seems like the right approach to me.  I didn't like the patches
posted by Quadrics a while ago, since they added hooks to all of
low-level page table hot paths.  Working at the vma level seems much
saner.

How much of a performance hit is this system call?  It seems that you
need to do the call before every single work request is posted, which
could cause latency to go way up.

    Pete> I can distribute the code if anyone is curious, but it needs
    Pete> some work to become production quality.

Yes, please post the code soon.

Thanks,
  Roland