[ofa-general] Re: [PATCH 1/11] IB/ipoib: high dma support

[Michael S. Tsirkin]

> Quoting Ralph Campbell <ralph.campbell at qlogic.com>:
> Subject: Re: [PATCH 1/11] IB/ipoib: high dma support
> 
> On Tue, 2007-09-25 at 16:41 +0200, Eli Cohen wrote:
> > On Tue, 2007-09-25 at 13:06 +0200, Or Gerlitz wrote:
> > > Eli Cohen wrote:
> > > > On Tue, 2007-09-25 at 12:22 +0200, Or Gerlitz wrote:
> > > >> Eli Cohen wrote:
> > > 
> > > >>> Add high dma support to ipoib
> > > >>> This patch assumes all IB devices support 64 bit DMA.
> > > 
> > > >> On some architectures DMA addresses are 32 bit, so I am not sure to 
> > > >> follow your comment. This capability states that the network device can 
> > > >> dma to high memory.
> > > 
> > > > I believe it means that *if* the kernel hands buffers whose addresses
> > > > exceed 32 bits then the device can handle them.
> > > 
> > > High-memory is well documents in books and elsewhere. I just want to say 
> > > that the change-log comment is confusing and unrelated.
> > > 
> > > What you want to say is that this patch assumes that for all IB devices, 
> > > ib_dma_map_single and ib_dma_map_page supports high memory, which is not 
> > > the case, see below.
> > > 
> > > Ralph?
> > > 
> > > Or.
> 
> Correct. ib_ipath doesn't support high memory and it would be
> inefficient to do so.
> 
> > > > static u64 ipath_dma_map_page(struct ib_device *dev,
> > > > 			      struct page *page,
> > > > 			      unsigned long offset,
> > > > 			      size_t size,
> > > > 			      enum dma_data_direction direction)
> > > > {
> > > > 	u64 addr;
> > > > 
> > > > 	BUG_ON(!valid_dma_direction(direction));
> > > > 
> > > > 	if (offset + size > PAGE_SIZE) {
> > > > 		addr = BAD_DMA_ADDRESS;
> > > > 		goto done;
> > > > 	}
> > > > 
> > > > 	addr = (u64) page_address(page);
> > > > 	if (addr)
> > > > 		addr += offset;
> > > > 	/* TODO: handle highmem pages */
> > > > 
> > > > done:
> > > > 	return addr;
> > > > }
> > > 
> > 
> > I got the impression that all supported IB devices support dma-ing
> > to/from memory > 4GB. Perhaps other vendors can comment.
> 
> The QLogic HCAs don't support DMA to or from the physical memory
> for the verbs Lkey/Rkey memory regions. The whole reason I added
> the ib_dma_*() functions was so to avoid ib_ipoib, etc. from
> calling dma_*() directly and passing a physical address as the
> offset in the posted work requests.
> What happens instead, is that ib_dma_*() returns a kernel virtual
> address which is passed in the work request and the driver copies
> the data to/from the HW as needed.
> So, in order to support HIGHMEM, I would need to change the
> ipath_dma_*() functions to call kmap()/kunmap() for HIGHMEM pages.
> I'm sure there would be all kinds of performance and coding issues
> around doing this.

So, we need some kind of HIGHDMA capability flag?

-- 
MST