[openib-general] How to support IOMMUs for ipath driver

[Ralph Campbell]

On Thu, 2006-09-14 at 13:51 +0300, Or Gerlitz wrote:
> Ralph Campbell wrote:
> > +static inline dma_addr_t ib_dma_map_sg(struct ib_device *dev,
> > +				       struct scatterlist *sg, int nents,
> > +				       enum dma_data_direction direction)
> > +{
> > +	return dev->map_sg ?
> > +		dev->map_sg(dev, sg, nents, direction) :
> > +		dma_map_sg(dev->dma_device, sg, nents, direction);
> > +}
> 
> As SG dma mapping happens in place and you don't want to change struct 
> scatterlist for every arch, i think you would need to keep some mapping 
> (hash) from each struct scatterlist to its ipath buddy...
> 
> Also you would need to implement the sg_dma_address() and sg_dma_len() 
> macros used by ULP code when page/s is/are to be input-ed for the IB 
> verbs layer eg to get an SG FMR-ed or send/recv from/into a page and use 
> queries into the ipath scatterlist buddy.
> 
> Or.

Here is my thinking so far:

The driver is passed an LKEY/RKEY plus an address.
For ib_get_dma_mr(), the address is currently from
dma_map_single(), dma_map_page(), or dma_map_sg().
With the ib_dma_*() routines, I can intercept these calls
and return something instead of a bus or IOMMU address.
I would like to return a kernel virtual address since that
is the simplest and is what I ultimately need. This is
trivial for dma_map_single() and trivial for low memory
pages for dma_map_page().

I think I can safely just return error for architectures
with high memory pages since the driver really only works
on 64-bit systems (for a variety of reasons which I won't
go into) and those systems don't have high memory.

If I did have to support high memory pages, I think the
DMA address would have to be the address of some kmalloc'ed
structure containing the page pointer and offset (or an
index into a table of such data structures).
I wouldn't want to make ib_dma_map_single() have to use that
but then I would need a way to distingush addresses
returned from ib_dma_map_single() and dma_map_page()/dma_map_sg().

ib_dma_map_sg() is a bit more complex.
The struct scatterlist is defined in the architecture specific headers.
The sg_dma_address() and sg_dma_len() macros are the exported
interface for accessing the DMA address and length.
I would like to minimize the impact to architecture specific code.
Given these constraints, I think the best thing to do is
add ib_sg_dma_address() and ib_sg_dma_len() functions which should
be used instead of sg_dma_address() and sg_dma_len().
Struct scatterlist has to contain at least page, offset, and length
since the SCSI code relies on those.
ib_sg_dma_address would return the page_address() of sg->page
but wouldn't be able to rely on other fields which might be in
the struct scatterlist.
Again, if high page support is needed, ib_sg_dma_address() would
have to do something trickier like for ib_dma_map_page().