[openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbsimplementation

[Fab Tillier]

> From: Bodo Eggert <harvested.in.lkml at posting.7eggert.dyndns.org>
> Sent: Friday, April 22, 2005 6:10 AM
> 
> All userspace hardware drivers with DMA will require pinned pages (and
> some of them will require continuous memory). Since this memory may be
> scheduled to be accessed by DMA, reclaiming those pages may (aka. will)
> result in "random" memory corruption unless done by the driver itself.

Any reclaim must involve the driver.  That doesn't mean that it must involve
the application.  That said this isn't trivial to implement.

> 
> You can't even set a time limit, the driver may have allocated all DMA
> memory to queued transfers, and some media needs to get plugged in by
> the lazy robot. As soon as the robot arrives - boom. (For the same reason,
> this memory MUST NOT be freed if the application terminates abnormally,
> e.g. killed by OOM).

InfiniBand provides support for deregistering memory that might be
referenced at some future time by an RDMA operation.  The only side effect
this has is that the QP on both sides of the connection transition to an
error state.

Upon abnormal termination, all registrations must be undone and the memory
unpinned.  This must be synchronized with the hardware so that there are no
races.  The IB deregistration semantics provide such synchronization.  I'd
venture that any HW design that does not do this is broken.

Requiring the memory to never be freed upon abnormal termination equates to
a serious memory leak, in that physical memory is leaked, not virtual.

- Fab