[openib-general] problem with 2.6.19?

[Steve Wise]

On Thu, 2006-10-26 at 15:26 -0700, Roland Dreier wrote:
>     Steve> The adapter seems to be dma'ing into the wrong memory.  The
>     Steve> patch below backs the usage of dma_map_single() back to
>     Steve> using __pa() for converting kernel virtual addresses (from
>     Steve> kmalloc) into bus addresses, and things work ok.
> 
> Hmm.  It might be interesting to hack the driver to print the result
> of both dma_map_single() and __pa() and see if they're different.
> 

Here's a dump.  They are different.  The __pa()'s are what I expect.  I
dunno if the dma_addr's returned from dma_map_single() look good or not
(they certainly don't work :)

c2_alloc_mqsp_chunk mqsp_chunk va ffff810148b1b000 dma_addr 3a56000 __pa 148b1b000
c2_alloc_mqsp addr ffff810148b1b01a dma_addr 3a5601a
c2_alloc_mqsp addr ffff810148b1b01c dma_addr 3a5601c
c2_alloc_mqsp addr ffff810148b1b01e dma_addr 3a5601e
c2_alloc_mqsp addr ffff810148b1b020 dma_addr 3a56020
c2_rnic_init rep_vq va ffff810147e78000 dma 3a57000 __pa 147e78000
c2_rnic_init aeq va ffff810147d48000 dma 3a5f000 __pa 147d48000

> Are you running on a 32-bit (i386) or 64-bit (x86_64) kernel?  How
> much RAM do you have?  

64b/X86_64.  4GB RAM.  The CPUs are Dempsey class XEONs - Dual CPU, Dual
core.  So with HT on linux sees 8 CPUs.

> Is the kernel using swiotlb?  If so then you
> need to make sure your DMA_{TO,FROM} directions and dma_unmap calls
> are right, since otherwise the DMAed data won't be copied to/from the
> bounce buffer at the right time.

All these mappings are for the device to DMA into the host memory, and
I'm using DMA_FROM_DEVICE in my calls to dma_map_single().

How do I know if the kernel is using swiotlb?


> Another thing to do if you're patient would be to use git-bisect and
> figure out exactly which patch made amso1100 stop working.
> 

I added these calls as part of the review for submission into the
kernel, and I originally tested them on dual CPU opteron systems with
1GB of memory.  But maybe they weren't using the IOMMU?  Dunno.