[openib-general] problem with 2.6.19?

[Roland Dreier]

 > I must be misusing dma_map_single().  What I'm doing is allocating a
 > verb message reply queue for the adapter to DMA verb replies into.  It
 > never gets unmapped. I kmalloc() it, then map it.  I could use
 > dma_alloc_coherent() or something, and maybe that's what I need to do?  

Yeah, if you want to leave something mapped and have the device DMA
into it, and the CPU look into the buffer too, then you need
consistent/coherent memory -- either pci_alloc_consistent() or
dma_alloc_coherent().  The dma_ variant is slightly better because you
can pass in a GFP_ mask rather than having the kernel pick GFP_ATOMIC
for you.

 > You're saying I must unmap it before the data is valid (cuz of the
 > bounce buffering).  If that's true, then how in sam hill does user mode
 > RDMA work since the user's memory isn't unmapped before the user looks
 > like memory that is the target of RDMA????  The uverbs code calls
 > dma_map_sg() which is roughly the same as dma_map_single, eh?

It's a good point.  We're kind of counting on the IOMMU situation not
being too wacky, and the device being able to DMA to arbitrary
addresses.  So swiotlb won't work in this case actually -- but we
assume any RDMA device can do 64-bit DMA so it doesn't hurt us in
practice.  But yes, DMA to userspace is slightly risky and won't work
everywhere.

 - R.