[openib-general] [PATCH 3/7] IB/ipoib - Use the new verbs DMA mapping functions

[Roland Dreier]

 > Hmm, since ib_dma_unmap_single calls a function through a pointer,
 > this seems to introduce overhead on data path operations in ipoib.
 > For apps like ipoib always working with low memory, I think it is important to avoid this
 > overhead of extra indirect function calls at least on systems without IO MMU -
 > where e.g. dma_unmap_single is empty.
 > This probably means you need some of architecture-dependent code,
 > but should be possible - look at how dma API is implemented for an example.
 > And this applies to all ULPs on systems without high memory.

How is this possible?  The IOMMU might be detected at runtime, and you
can always have a system with multiple HCAs of different types, so I
don't see how the conditional can be avoided.  It is unfortunate but
in this case I think we have to accept the cost of making the code
general.

It is sad that ipath is likely the only driver that will ever use
this.  Maybe something that the speed-freaks would like would be to
add a hidden config option that turns all the ib_dma_xxx stuff into
NOP macros unless ipath is being built.  Of course that doesn't help
all that much because all the distros etc will enable ipath.

Anyway, I suspect the penalty is near-zero anyway, since the pointer
being tested will likely be in cache and the branch predictor will
learn which way the branch goes.  (Except on a heterogeneous system I
suppose)

 - R.