[openib-general] NFS/RDMA client release for Linux 2.6.15

[Talpey, Thomas]

At 05:28 PM 2/19/2006, Roland Dreier wrote:
>    Christoph> - rdma_convert_physiov/rdma_convert_phys are completely
>    Christoph> broken.  page_to_phys can't be used by driver/fs code.
>    Christoph> RDMA only deals with bus addresses, not physical
>    Christoph> addresses.  You must use the dma mapping API
>    Christoph> instead. Also coalescing decisions are made by the dma
>    Christoph> layer, because they are platform dependent and much
>    Christoph> more complex then what the code in this patch does.
>
>    Thomas> Now that we are moving to OpenIB api's this is
>    Thomas> needed. There is some thought necessary w.r.t. our
>    Thomas> max-performance mode of preregistering memory in DMA
>    Thomas> mode. That's on our list of course.
>
>Again let me echo Christoph's point.  If you are passing physical
>addresses into IB functions, then your code simply won't work on some
>architectures.  Making sure your code actually works on something like
>a ppc64 box with an IOMMU would be a good test -- the low-end IBM
>POWER machines are cheap enough that you could just buy one if you
>don't have easy access.

Yep, I get it!

To elaborate a little, we're not exactly passing physical addresses. What
we're doing is using the physaddr to calculate an offset relative to a base
of zero. We register the zero address and advertise RDMA buffers via
offsets relative to that r_key.

And, this is only one of many memory registration modes. We would use
memory windows, if only OpenIB provided them (yes I know the hardware
currently sucks for them). We will add FMR support shortly. In both these
modes we perform all addressing by the book via 1-1 OpenIB registration.

Tom.