[openib-general] problem with 2.6.19?

[Steve Wise]

So the issue is my maps are getting setup with bounce bufs.  This
shouldn't be the case, I think, because my device supports 64b
addressing.  I'm diggin' into this.  



On Fri, 2006-10-27 at 10:53 -0500, Steve Wise wrote:
> And I do have CONFIG_SWIOTLB set:
> 
> CONFIG_SWIOTLB=y
> 
> On Fri, 2006-10-27 at 10:48 -0500, Steve Wise wrote:
> > Also, if I drop the memory down to 1G (via mem=1G on the boot line),
> > then things work.  But I think I'm just disabling the IOMMU.  Note that
> > __pa() and dma_map_single() return the same thing:
> > 
> > 
> > c2_alloc_mqsp_chunk mqsp_chunk va ffff81003c4d2000 dma_addr 3c4d2000 __pa 3c4d2000
> > c2_alloc_mqsp addr ffff81003c4d201a dma_addr 3c4d201a
> > c2_alloc_mqsp addr ffff81003c4d201c dma_addr 3c4d201c
> > c2_alloc_mqsp addr ffff81003c4d201e dma_addr 3c4d201e
> > c2_alloc_mqsp addr ffff81003c4d2020 dma_addr 3c4d2020
> > c2_rnic_init rep_vq va ffff81003cb08000 dma 3cb08000 __pa 3cb08000
> > c2_rnic_init aeq va ffff81003d4e0000 dma 3d4e0000 __pa 3d4e0000
> > 
> > 
> > 
> > On Fri, 2006-10-27 at 09:43 -0500, Steve Wise wrote:
> > > On Thu, 2006-10-26 at 15:26 -0700, Roland Dreier wrote:
> > > >     Steve> The adapter seems to be dma'ing into the wrong memory.  The
> > > >     Steve> patch below backs the usage of dma_map_single() back to
> > > >     Steve> using __pa() for converting kernel virtual addresses (from
> > > >     Steve> kmalloc) into bus addresses, and things work ok.
> > > > 
> > > > Hmm.  It might be interesting to hack the driver to print the result
> > > > of both dma_map_single() and __pa() and see if they're different.
> > > > 
> > > 
> > > Here's a dump.  They are different.  The __pa()'s are what I expect.  I
> > > dunno if the dma_addr's returned from dma_map_single() look good or not
> > > (they certainly don't work :)
> > > 
> > > c2_alloc_mqsp_chunk mqsp_chunk va ffff810148b1b000 dma_addr 3a56000 __pa 148b1b000
> > > c2_alloc_mqsp addr ffff810148b1b01a dma_addr 3a5601a
> > > c2_alloc_mqsp addr ffff810148b1b01c dma_addr 3a5601c
> > > c2_alloc_mqsp addr ffff810148b1b01e dma_addr 3a5601e
> > > c2_alloc_mqsp addr ffff810148b1b020 dma_addr 3a56020
> > > c2_rnic_init rep_vq va ffff810147e78000 dma 3a57000 __pa 147e78000
> > > c2_rnic_init aeq va ffff810147d48000 dma 3a5f000 __pa 147d48000
> > > 
> > > > Are you running on a 32-bit (i386) or 64-bit (x86_64) kernel?  How
> > > > much RAM do you have?  
> > > 
> > > 64b/X86_64.  4GB RAM.  The CPUs are Dempsey class XEONs - Dual CPU, Dual
> > > core.  So with HT on linux sees 8 CPUs.
> > > 
> > > > Is the kernel using swiotlb?  If so then you
> > > > need to make sure your DMA_{TO,FROM} directions and dma_unmap calls
> > > > are right, since otherwise the DMAed data won't be copied to/from the
> > > > bounce buffer at the right time.
> > > 
> > > All these mappings are for the device to DMA into the host memory, and
> > > I'm using DMA_FROM_DEVICE in my calls to dma_map_single().
> > > 
> > > How do I know if the kernel is using swiotlb?
> > > 
> > > 
> > > > Another thing to do if you're patient would be to use git-bisect and
> > > > figure out exactly which patch made amso1100 stop working.
> > > > 
> > > 
> > > I added these calls as part of the review for submission into the
> > > kernel, and I originally tested them on dual CPU opteron systems with
> > > 1GB of memory.  But maybe they weren't using the IOMMU?  Dunno.
> > > 
> > > 
> > > 
> > > 
> > > _______________________________________________
> > > openib-general mailing list
> > > openib-general at openib.org
> > > http://openib.org/mailman/listinfo/openib-general
> > > 
> > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> > > 
> > 
> > 
> > _______________________________________________
> > openib-general mailing list
> > openib-general at openib.org
> > http://openib.org/mailman/listinfo/openib-general
> > 
> > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> > 
> 
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>