[openib-general] [PATCH] kDAPL: cleanup dat/ a bit more

[Caitlin Bestler]

On 6/6/05, Grant Grundler <iod00d at hp.com> wrote:
> On Sat, Jun 04, 2005 at 04:33:12PM -0700, Caitlin Bestler wrote:
> ...
> > The Quadrics patch informed zero copy devices
> > of new mappings, but does so when the device
> > is not actively doing work. That latter piece of
> > information is not available for an RDMA device.
> 
> It certainly is.
> 

As your reply indicates below, the OS does *not*
know *when* the device is active. The memory
is registered as being device accessible for an
indefinite period of time (must likely until the 
machine reboots).

The information on when an RDMA device is
*active* is NOT available to the OS.

> Anyway, I agree with Christoph.
> 
> > For normal devices the OS has a reasonable
> > expectation on when the device is performing
> > DMA, because it is doing so in response to a
> > request that the OS knows about. For an RDMA
> > device a DMA transfer can be in progress as
> > as a result of an RDMA Read -- something the
> > host processor knows nothing about.
> 
> That's definitely not true. Host maps all DMA.
> No one can DMA to/from the host without registering the
> memory for a DMA mapping. The host does NOT know when
> any given DMA mapping is active - but that's true for
> FC/SCSI devices today too.
> 
> The device driver must take reasonable measures
> to quiesce that DMA mapping.  Maybe send a signal
> to the task that owns the DMA mapping?
> Ie warn the task, wait a bit, then kill the task with
> extreme prejudice if the resources it's sitting on
> aren't free yet. I don't expect Hotplug support
> to be simple to implement. But it's definitely possible.
> 

Killing the task will not promptly cause RDMA activity
to cease. The memory region must be deallocated or
the RDMA device handle must be released. So you're
back to how the kernel verbs are told of this.

In any event, what you describe would only work for
shutdown. It would not work to migrate pages. The
same interface can be used for both purposes.