[openib-general] [PATCH] kDAPL: cleanup dat/ a bit more

[Caitlin Bestler]

On 6/4/05, Christoph Hellwig <hch at lst.de> wrote:

> 
> > That is
> > why the Quadrics extensions for zero placement are not
> > applicable to an RDMA device. That extension, for example,
> > only remaps memory when the IO device is not actively
> > doing a transfer. Something that the kernel does not
> > know with an RDMA device.
> 
> How is memory remapping related to device removal?
> 
> 

Different problem, but the required solutions overlap.

To remap a pinned memory region the OS must tell
the RDMA device to suspend it's access to that memory.
In doing so it must wait for in-progress transfers to complete.

The RDMA Device is then in a state where it defers all
activities associated with the suspended memory.
Work Reqeusts that reference it are effectively fenced.
Inbound packets that refernce it must be deferred
(by dropping it, RNRs or interim buffering).

After the OS has migrated the pages it supplies
the new mapping to the RDMA Device, which lets
it resume the Memory Region (after it has updated
the PBLEs).

The Quadrics patch informed zero copy devices
of new mappings, but does so when the device
is not actively doing work. That latter piece of
information is not available for an RDMA device.
For normal devices the OS has a reasonable
expectation on when the device is performing
DMA, because it is doing so in response to a
request that the OS knows about. For an RDMA
device a DMA transfer can be in progress as
as a result of an RDMA Read -- something the
host processor knows nothing about.