[openib-general] [kernel verbs] u64 vs dma_addr_t

[James Lentini]

On Fri, 30 Dec 2005, Caitlin Bestler wrote:

caitlin> On 12/30/05, James Lentini <jlentini at netapp.com> wrote:
caitlin> >
caitlin> >
caitlin> > caitlin> > One more question on this topic.
caitlin> > caitlin> >
caitlin> > caitlin> > Why is the ib_sge's addr a u64 and not a dma_addr_t?
caitlin> > caitlin>
caitlin> > caitlin> Because the hardware may need for it to be a 64 bit
caitlin> > caitlin> IO Address accessible on the system bus. That applies
caitlin> > caitlin> to the whole system bus, no matter how many PCI roots
caitlin> > caitlin> or virtual OSs there are.
caitlin> > caitlin>
caitlin> > caitlin> In particular there could be a guest OS that was
caitlin> > caitlin> running in 32-bit mode, and the RDMA hardware receiving
caitlin> > caitlin> fast path requests will not support different
caitlin> > caitlin> work request formats for each guest OS.
caitlin> >
caitlin> > Let me back up a step and explain the context for this question.
caitlin> >
caitlin> > As you know, our goal is to use the Linux IB verbs as a
caitlin> > hardware/protocol independent RDMA API. I'm reviewing my use of the
caitlin> > API to make sure that it does not make any particular assumptions.
caitlin> >
caitlin> > Roland stated that a scatter/gather list's address value should be a
caitlin> > bus address:
caitlin> >
caitlin> > http://openib.org/pipermail/openib-general/2005-August/009748.html
caitlin> >
caitlin> 
caitlin> That depends on whether it is part of a registered memory space,
caitlin> or being used to specify a new registered memory space (i.e. it is
caitlin> for a memory register operation).

That is not an issue. The memory registration functions 
(ib_reg_phys_mr(), etc.) describe memory regions using a ib_phys_buf 
structure.

caitlin> When *using* an already established memory region, the address
caitlin> is interpreted in the context of that memory region. The size of
caitlin> address within an RDMA managed memory regions is always
caitlin> 64 bits. No matter which transport or what processor. That is
caitlin> extremely unlikely to change (in fact I think the R-Key/L-Key/
caitlin> STag size would increase to 64-bits before the address size
caitlin> itself changed. But I'm expecting that a 96-bit logical address
caitlin> space should be adequate for quite some time).
caitlin> 
caitlin> When creating a memory region the "physical address" is 
caitlin> really a bus address, which on a strictly local basis could 
caitlin> be 32 or 64 bits. If you were trying to generalize that, the 
caitlin> "physical address" is a "RDMA Device accessible address", 
caitlin> which on anything even vaguely PCI-ish is a bus address.
caitlin> 
caitlin> But just as the distinction between "physical address" and 
caitlin> "bus address" would not have been anticipated in the past, 
caitlin> there may be some other distinction that we are not 
caitlin> anticipating yet. So, in that context, the Memory Region 
caitlin> defines the translation from logical addresses with the 
caitlin> context of a Memory Region (most typically a subset of an 
caitlin> existing virtual memory map) to addresses that the RDMA 
caitlin> device can use to access the same memory. Whatever that 
caitlin> distinction is, I'm sure it will be relevant before another 
caitlin> decade goes by.

Are you arguing that an ib_sge addr should be a u64 or a dma_addr_t?