[openib-general] [PATCH v2 1/7] IB/core - Add DMA mapping functions to allow device drivers to interpose

[Ralph Campbell]

On Tue, 2006-12-05 at 23:21 +0200, Or Gerlitz wrote:
> On 12/5/06, Roland Dreier <rdreier at cisco.com> wrote:
> > I think this seems reasonable.  And I think it also provides a way to
> > address some hypothetical future situation where lowmem pages don't
> > have a kernel virtual address -- you would just have to use this
> > type of cookie implementation everywhere.
> 
> Such an approach would be much more cleaner and result in much less
> (~zero changes) in the ulp level, just replace dma_map_xxx calls with
> ib_dma_map_xxx calls.
> 
> A problem  see with the dma_addr_t being a cookie into a table of kv
> addresses is that its legal for a consumer to use dma_addr_t with an
> **offset** . So she gets addr y from ib_dma_map_xxx and then uses y +
> offset in the SGE provided to ibv_post_send/recv or to the fmr map
> function.
> 
> So this table is actually a search tree which allows you to match an
> offset-ed dma_addr_t returned by dma_map_xxx called by ipath
> ib_dma_map_xxx with its associated kvaddr.
> 
> I see now that i have managed to confuse myself b/c as Roland wrote
> below and i have agreed we don't actually have the kv addr for and
> unmapped page before the ipath driver maps it ie when it attempt to
> use the page... It becomes late here... am i inventing a non existant
> problem with the offset?
> 
> > (Although I don't think using kmap()/kunmap() is really the right
> > approach -- you should probably just do kmap_atomic()/kunmap_atomic()
> > only while you are actually using the page.  But the basic approach of
> > using the dma address as a cookie into a mapping table seems sound to
> > me -- you are basically doing a real sw iotlb)
> >
> > Or -- does this seem reasonable to you?
> 
> I agree that care should be made to do kmap_atomic/kunmap_atomic only
> when there is actual need to access the page by the ipath driver.
> 
> Or.

I am not following what you two are saying.

The ib_dma_mapping_ops functions as implemented by ib_ipath,
are redefining dma_addr_t as a kernel virtual address.
When ib_dma_map_single() is called, this is a NOP.
When ib_dma_map_sg() is called, the dma_map_sg() replacement needs
to convert a struct page pointer into a kernel virtual address.
When CONFIG_HIGHMEM is defined, some pages may not be mapped
into the kernel virtual address space so the driver needs to
call kmap().  Since the driver can't use the struct scattergather
to store the kmap() result, a separate table needs to be used
so the value can be returned by ib_sg_dma_address().

Doing kmap_atomic() at the point where the kernel virtual
address is used is not practical since the driver is not
mapping dma_addr_t to struct page * although it is
possible to write it that way.  It would mean that
ib_map_single() would then be more complex in that a
kernel virtual address would need to be converted to a
struct page *.