[openib-general] Re: [PATCH 5 of 13] ipath - use proper address translation routine

[Caitlin Bestler]

openib-general-bounces at openib.org wrote:
> proper address translation routine
> 
> On Tue, May 02, 2006 at 07:24:18AM -0700, Roland Dreier wrote:
>>     Christoph> Or stop doing the dma mapping in the IB upper level
>>     Christoph> drivers.  I told you that we'll get broken hardware
>>     Christoph> that doesn't want dma mapping in the upper level
>>     Christoph> driver, and pathscale created exactly that :)
>> 
>> But see my earlier mail to Arjan about RDMA -- what address can a
>> protocol (eg SRP initiator) put in a message that the other side will
>> use to initiate a remote DMA operation?  It seems to me it has to be
>> a bus address, and that means that the protocol has to do the DMA
>> mapping. 
> 
> Then we're back to the discussion on why RDMA is a
> fundamentally flawed approach, but we already knew that.  The
> usual workaround is to only allow RDMA operations to
> registered memory windows for which we can use the normal dma
> operation.  There's also the *dac* pci dma operations that
> can avoid iommu overhead if you support 64bit addressing.
> But for all this to work dma mapping fundamentally needs to
> be handled by the low level driver.

This is not a flaw in the RDMA model. All the RDMA Model requires
is exposing virtual addresses that the device can translate back
to host memory for remote operations. The protocols do not specify
what the backing of an R-Key or STag is.

And the local interface only requires that a L-Key / MR Stag be
backed by translations from Key/TAG and Address to a memory
reference that the hardware is capable of using immediately.

So the problem here is that the upper layer driver does not properly
understand what type of address the specific driver requires (and/or
of the driver/device to understand what type of addresses it will be
given). That's a failure to document interface requirements, not a
failure of the RDMA model.