[openib-general] [kernel verbs] u64 vs dma_addr_t

[Caitlin Bestler]

James Lentini wrote:
> On Fri, 30 Dec 2005, Caitlin Bestler wrote:
> 
>> openib-general-bounces at openib.org wrote:
>>> sean> James Lentini wrote:
>>> sean> > Why is the ib_sge's addr a u64 and not a dma_addr_t? sean>
>>> sean> It's the same address that the user can transfer to the
>>> remote sean> side. 
>>> 
>>> It can be the same address, but does it have to be?
>>> 
>>> A user can directly map local addresses to InfiniBand I/O virtual
>>> addresses, but I don't think it is a requirement. In other words, I
>>> thought that user could register address x and request an InfiniBand
>>> I/O virtual address of y, x != y, for the mapping.
>>> 
>>> I understand why the ib_send_wr's rdma.remote_addr needs to be a
>>> u64, since it ultimately winds up on the wire.
>>> 
>>> In the case of the ib_sge's addr, I didn't think these values left
>>> the local node. My assumption (based on looking at the mthca driver)
>>> is that they are supposed to contain "local"
>>> I/O addresses (bus addresses). Therefore, my confusion over why
>>> dma_addr_t wasn't used.
>> 
>> A privileged user, such as an NFS Daemon or iSER iSCSI Target, can
>> and will create Memory Regions that are not part of its own address
>> space out of page buffers. Even running on a 32-bit kernel it might
>> create a memory region larger than 2**32.
>> 
>> Admittedly, that isn't very likely unless it is the *only* daemon
>> running on the machine. But it is legal.
> 
> How does the size of a region relate to my original question
> on the type of an ib_sge.addr?

The address has to be big enough to enumerate all bytes in the region.

If a region is not constrained to fit within an existing virtual
memory map then it is not constrained by the size of a virtual address,
only by the size of MRs and of physical addresses.