[ofa-general] Re: Demand paging for memory regions

[Christoph Lameter]

On Fri, 15 Feb 2008, Caitlin Bestler wrote:

> So that would mean that mlock is used by the application before it 
> registers memory for direct access, and then it is up to the RDMA
> layer and the OS to negotiate actual pinning of the addresses for
> whatever duration is required.

Right.
 
> There is no *protocol* barrier to replacing pages within a Memory
> Region as long as it is done in a way that keeps the content of
> those page coherent. But existing devices have their own ideas
> on how this is done and existing devices are notoriously poor at
> learning new tricks.

Hmmmm.. Okay. But that is mainly a device driver maintenance issue.

> Merely mlocking pages deals with the end-to-end RDMA semantics.
> What still needs to be addressed is how a fastpath interface
> would dynamically pin and unpin. Yielding pins for short-term
> suspensions (and flushing cached translations) deals with the
> rest. Understanding the range of support that existing devices
> could provide with software updates would be the next step if
> you wanted to pursue this.

That is addressed on the VM level by the mmu_notifier which started this 
whole thread. The RDMA layers need to subscribe to this notifier and then 
do whatever the hardware requires to unpin and pin memory. I can only go 
as far as dealing with the VM layer. If you have any issues there I'd be 
glad to help.