[ofa-general] [PATCH 1/11] IB/ipoib: high dma support

[Ralph Campbell]

On Tue, 2007-09-25 at 16:41 +0200, Eli Cohen wrote:
> On Tue, 2007-09-25 at 13:06 +0200, Or Gerlitz wrote:
> > Eli Cohen wrote:
> > > On Tue, 2007-09-25 at 12:22 +0200, Or Gerlitz wrote:
> > >> Eli Cohen wrote:
> > 
> > >>> Add high dma support to ipoib
> > >>> This patch assumes all IB devices support 64 bit DMA.
> > 
> > >> On some architectures DMA addresses are 32 bit, so I am not sure to 
> > >> follow your comment. This capability states that the network device can 
> > >> dma to high memory.
> > 
> > > I believe it means that *if* the kernel hands buffers whose addresses
> > > exceed 32 bits then the device can handle them.
> > 
> > High-memory is well documents in books and elsewhere. I just want to say 
> > that the change-log comment is confusing and unrelated.
> > 
> > What you want to say is that this patch assumes that for all IB devices, 
> > ib_dma_map_single and ib_dma_map_page supports high memory, which is not 
> > the case, see below.
> > 
> > Ralph?
> > 
> > Or.

Correct. ib_ipath doesn't support high memory and it would be
inefficient to do so.

> > > static u64 ipath_dma_map_page(struct ib_device *dev,
> > > 			      struct page *page,
> > > 			      unsigned long offset,
> > > 			      size_t size,
> > > 			      enum dma_data_direction direction)
> > > {
> > > 	u64 addr;
> > > 
> > > 	BUG_ON(!valid_dma_direction(direction));
> > > 
> > > 	if (offset + size > PAGE_SIZE) {
> > > 		addr = BAD_DMA_ADDRESS;
> > > 		goto done;
> > > 	}
> > > 
> > > 	addr = (u64) page_address(page);
> > > 	if (addr)
> > > 		addr += offset;
> > > 	/* TODO: handle highmem pages */
> > > 
> > > done:
> > > 	return addr;
> > > }
> > 
> 
> I got the impression that all supported IB devices support dma-ing
> to/from memory > 4GB. Perhaps other vendors can comment.

The QLogic HCAs don't support DMA to or from the physical memory
for the verbs Lkey/Rkey memory regions. The whole reason I added
the ib_dma_*() functions was so to avoid ib_ipoib, etc. from
calling dma_*() directly and passing a physical address as the
offset in the posted work requests.
What happens instead, is that ib_dma_*() returns a kernel virtual
address which is passed in the work request and the driver copies
the data to/from the HW as needed.
So, in order to support HIGHMEM, I would need to change the
ipath_dma_*() functions to call kmap()/kunmap() for HIGHMEM pages.
I'm sure there would be all kinds of performance and coding issues
around doing this.