[openib-general] [kernel verbs] u64 vs dma_addr_t

[James Lentini]

sean> James Lentini wrote:
sean> > Why is the ib_sge's addr a u64 and not a dma_addr_t? 
sean> 
sean> It's the same address that the user can transfer to the remote 
sean> side. 

It can be the same address, but does it have to be?

A user can directly map local addresses to InfiniBand I/O virtual 
addresses, but I don't think it is a requirement. In other words, I 
thought that user could register address x and request an InfiniBand 
I/O virtual address of y, x != y, for the mapping.

I understand why the ib_send_wr's rdma.remote_addr needs to be a u64, 
since it ultimately winds up on the wire.

In the case of the ib_sge's addr, I didn't think these values left the 
local node. My assumption (based on looking at the mthca driver) is 
that they are supposed to contain "local" I/O addresses (bus 
addresses). Therefore, my confusion over why dma_addr_t wasn't used.

sean> Also, if inline sends are being used, the address is not 
sean> necessarily a DMA address.

Which ib_wr_opcode[s] are "inline sends"? IB_WR_SEND, 
IB_WR_SEND_WITH_IMM, ...? 

My expectation was that all of the scatter/gather data for both sends 
(of all flavors: send, rdma read, rdma write,...) and recvs would be 
DMA.

The "not necessarily" part makes me worry. How can I determine in a 
device independent way which buffer need to be DMA'able and which do 
not? Is the only safe option to assume that all do?