[openib-general] How to support IOMMUs for ipath driver

[Ralph Campbell]

On Thu, 2006-09-14 at 09:36 +0300, Or Gerlitz wrote:
> Ralph Campbell wrote:
> > On Wed, 2006-09-13 at 12:00 +0300, Or Gerlitz wrote:
> >> Ralph Campbell wrote:
> 
> > Well, the other parts of the kernel might not need a kernel virtual
> > address but the ib_ipath driver still does.
> 
> So you agree there is a need to kmap/kunamp pages which the user wants 
> to  use with IB and are not mapped into the kernel virt address space?

Yes, I agree for systems which have high memory pages.

> > I don't understand what you are talking about. There is an IB
> > wire protocol for RDMA, SEND, etc. That doesn't change depending
> > on the HCA.
> > The InfiniPath HCA has a ring buffer of receive buffers and all
> > incoming IB packets are DMA'ed into one of these buffers.
> > The ib_ipath software driver examines the packet and
> > copies it to the appropriate address. For a packet received with
> > a RC_RDMA_WRITE_FIRST, the RKEY and IB address are used to convert
> > that into a kernel virtual address and the data is copied.
> > The same happens for RC_SEND_FIRST but the KV address comes from
> > the LKEY and address in the work request posted by ib_post_recv().
> 
> OK, this make sense.
> 
> Lets see if i follow: you say that the Infinipath HCA is RX DMA-able but 
> it does RX DMA to the ipath driver private RX buffers and then the 
> driver copies from these buffers to the user buffer. My guess is that 
> you do that to support both recv and rdma read on this QP since if you 
> would only need to support recv you can have the hca dma-ing to the user 
> posted rx buffer.

You mostly understand. The hardware doesn't have separate receive
queues for each QP. All packets go into a single (or at most 4
currently) receive queues and the driver figures out which QP,
RDMA memory region, etc. to copy them to.