[openib-general] How to support IOMMUs for ipath driver

[Michael S. Tsirkin]

Quoting r. Ralph Campbell <ralphc at pathscale.com>:
> Subject: How to support IOMMUs for ipath driver
> 
> Problem:
> 
> The IB kernel to IB device driver interface uses dma_map_single()
> and dma_map_sg() to allocate device bus addresses for HW DMA.
> These bus addresses are passed to the IB device driver via ib_post_send()
> and ib_post_recv().
> 
> The ib_ipath driver needs kernel virtual addresses in order to be able
> to copy data to/from the posted work requests since it does not
> use HW DMA. It currently relies on the mapping being one-to-one
> and cannot reasonably reverse the mapping when an IOMMU is present.

Actually, Mellanox HCAs support DMA, but they *also* have the ability to copy
data to the posted work requests - this is the INLINE flag which we only
implemented in userspace - but not in kernel, since we could not get at the
kernel virtual address. It is actually useful for reducing latency for small
messages.

I wander whether ehca also can benefit from this capability. Anyone?

Unfortunately, the API you propose is not flexible enough in this respect
as it still does not seem to allow this optimization in kernel.
If we are changing the API, I would like the new API to be flexible
enough to enable this optimization.

I am not exactly sure what the best way to do that would be.
How about making it possible for ULPs to pass in kernel virtual address
in post send?

-- 
MST