[openib-general] [PATCH 1/7] IB/core - Add DMA mapping functions to allow device drivers to interpose

[Ralph Campbell]

Now that I'm back from vacation, I am answering my emails...

On Tue, 2006-11-14 at 09:49 +0200, Or Gerlitz wrote:
> Ralph Campbell wrote:
> > IB/core - Add DMA mapping functions to allow device drivers to interpose
> > 
> > The QLogic InfiniPath HCAs use programmed I/O instead of HW DMA.
> > This patch allows a verbs device driver to interpose on DMA mapping
> > function calls in order to avoid relying on bus_to_virt() and
> > phys_to_virt() to undo the mappings created by dma_map_single(),
> > dma_map_sg(), etc.
> 
> Yet another issue i see here: i understand that the core you are using 
> at your HCA is capable of doing DMA but is able to do so from one RX and 
> one TX cyclic descriptor buffer, correct? (*)

Close. You are correct for RX but for TX, the CPU copies the data to
the chip directly.

> Hence on the RX flow, you first get the IB packet into a buffer 
> allocated by the driver and then copy it to the buffer posted by the 
> user to the QP. But what is going on the TX flow, per my understanding 
> here you should be able to do it zero-copy wise but you do need to 
> provide a **dma** (and not kernel virtual nor physical) address to the HCA.
> 
> This means that for DMA_TO_DEVICE mappings you do want to call the 
> original dma_map_xxx calls and Roland's suggestion to have 
> ib_dma_map_xxx calls return u64 is problematic from another aspect then 
> the one i have pointed on.

I will need to think about dma_addr_t being 32-bit on sparc.
Perhaps Roland's suggestion to return u64 is OK but then there
is the issue of where to put the extra bits for ib_dma_map_sg().

> Going to the SC06 and then vacationing, I will probably not be able to 
> further jump on this thread before Nov 26th, anyway, my personal feeling 
> is that this need to be baked a little more, specifically pushing it for 
> 2.6.20-rc1 might be moving too fast... but if you guys feel the open 
> issues are resolved feel free to go ahead.
> 
> cheers,
> 
> Or.
> 
> (*) kernel bypass for rx/tx postings/completions by libipath seems to me 
> as black magic... you have to copy the rx data from the driver cyclic 
> buffer to the buffer pointed by tge user space qp wqe without passing 
> through the kernel. Do you do it on the poll_cq time? how do you post tx?

User level kernel bypass is only done for post receive and poll
currently. Send uses a system call.