[openib-general] [kernel verbs] u64 vs dma_addr_t

[Caitlin Bestler]

On 12/30/05, James Lentini <jlentini at netapp.com> wrote:
>
>
> caitlin> > One more question on this topic.
> caitlin> >
> caitlin> > Why is the ib_sge's addr a u64 and not a dma_addr_t?
> caitlin>
> caitlin> Because the hardware may need for it to be a 64 bit
> caitlin> IO Address accessible on the system bus. That applies
> caitlin> to the whole system bus, no matter how many PCI roots
> caitlin> or virtual OSs there are.
> caitlin>
> caitlin> In particular there could be a guest OS that was
> caitlin> running in 32-bit mode, and the RDMA hardware receiving
> caitlin> fast path requests will not support different
> caitlin> work request formats for each guest OS.
>
> Let me back up a step and explain the context for this question.
>
> As you know, our goal is to use the Linux IB verbs as a
> hardware/protocol independent RDMA API. I'm reviewing my use of the
> API to make sure that it does not make any particular assumptions.
>
> Roland stated that a scatter/gather list's address value should be a
> bus address:
>
> http://openib.org/pipermail/openib-general/2005-August/009748.html
>

That depends on whether it is part of a registered memory space,
or being used to specify a new registered memory space (i.e. it is
for a memory register operation).

When *using* an already established memory region, the address
is interpreted in the context of that memory region. The size of
address within an RDMA managed memory regions is always
64 bits. No matter which transport or what processor. That is
extremely unlikely to change (in fact I think the R-Key/L-Key/
STag size would increase to 64-bits before the address size
itself changed. But I'm expecting that a 96-bit logical address
space should be adequate for quite some time).

When creating a memory region the "physical address" is
really a bus address, which on a strictly local basis could be
32 or 64 bits. If you were trying to generalize that, the "physical
address" is a "RDMA Device accessible address", which on
anything even vaguely PCI-ish is a bus address.

But just as the distinction between "physical address" and
"bus address" would not have been anticipated in the past,
there may be some other distinction that we are not anticipating
yet. So, in that context, the Memory Region defines the translation
from logical addresses with the context of a Memory Region
(most typically a subset of an existing virtual memory map)
to addresses that the RDMA device can use to access the
same memory. Whatever that distinction is, I'm sure it will
be relevant before another decade goes by.