[ofa-general] Re: Demand paging for memory regions

[Caitlin Bestler]

On Feb 13, 2008 3:02 PM, Christoph Lameter <clameter at sgi.com> wrote:

>
> The main use of the notifier scheme is for stability and reliability. The
> "pinned" pages become unpinnable on request by the VM. So the VM can work
> itself out of memory shortage situations in cooperation with the
> RDMA logic instead of simply failing.
>

The very limited objective presented above was actually discussed in RNIC-PI.
A minimalist solution (from the hardware viewpoint) is to "suspend" a Memory
Region for a very brief time to allow the Host to re-arrange memory, and then
to "resume" operation once the pages were copied and the map updated.

The RDMA device has to avoid processing incoming packets that reference
the suspended Memory Region (rather than failing the connection) and flush
any cached mappings from before the "suspend" so that everything is learned/
fetched after the "resume".

The advertised pages have to have the same *meaning* and they have to be
committed, but they do not have to be the same physical pages for the
lifetime of the memory region (at least from the protocol perspective).

Obviously any add-on hardware functionality would have to be a documented
option so that  the memory manager would know whether a given device
actually could do this.