[openib-general] DMA mapping abuses in MAD layer

[Sean Hefty]

>properly.  Once we call dma_map_single() on a buffer, the CPU may not
>touch that buffer until after the corresponding dma_unmap_single().

It sounds like we need to change how the mapping is done.  Can we let the MAD
layer always control the mapping?  Considering how RMPP works, I'm not sure what
else we could do.

>is just a cludge -- the real fix will need to be more invasive.  It
>seems that the whole interface to the MAD layer may need to be
>reorganized to avoid doing this.

We really just need to change the post_send_mad routine, don't we?

The original intent around that API was to permit posting the WR directly onto
the QP.  Since this isn't the case, what about changing post send to take as
input an ib_mad_send_buf, with the work request and SGE fields removed?  This
could permit some additional optimization, such as avoiding additional
allocations within the post send call.  (Taking it a step further, we could
create a new structure to permit using a received MAD as input to a send.)

>It looks like there is a similar problem with ib_create_send_mad(): it
>does DMA mapping on a buffer that is then returned for the caller to modify.

If we pass the send_buf into post_send_mad, then the mapping could be deferred.

>Finally, some of the MAD structures like struct ib_mad_private look
>risky to me, since kernel data might potentially share a cache line
>with DMA buffers.  See <http://lwn.net/Articles/2265/> for a nice
>writeup of the class of bug that might be lurking.

This sounds like a separate issue, is that the case?

- Sean