[openib-general] DMA mapping abuses in MAD layer

[Fab Tillier]

> From: Roland Dreier [mailto:rolandd at cisco.com]
> Sent: Tuesday, October 11, 2005 8:39 PM
> 
> On mainstream architectures, it turns out that we can get away with
> violating this rule.  However, on non-cache-coherent architectures
> like PowerPC 4xx, dma_map_single(..., DMA_TO_DEVICE) does a cache
> flush, which makes sure that the contents of the CPU's cache are
> really written to memory.  If a driver then changes the contents of
> the buffer after the call to dma_map_single(), then it's quite likely
> that the change will be made only in the CPU's cache and the device
> will end up DMA-ing the old data.
> 
> The problem I hit is in ib_post_send_mad(), specifically:
> 
> 		smp = (struct ib_smp *)send_wr->wr.ud.mad_hdr;
> 		if (smp->mgmt_class == IB_MGMT_CLASS_SUBN_DIRECTED_ROUTE) {
> 			ret = handle_outgoing_dr_smp(mad_agent_priv, smp,
> 						     send_wr);
> 
> basically, when the MAD layer goes to send a directed route reply, it
> changes the MAD buffer after the DMA mapping is done.  The HCA
> doesn't see the change, the wrong packet gets sent and the SM never
> sees replies to its queries.
> 
> Adding a PPC-specific cache flush call after the call to
> handle_outgoing_dr_smp() fixes things to the point that the port can
> be brought to ACTIVE, and in fact IPoIB works as well.  However, this
> is just a cludge -- the real fix will need to be more invasive.  It
> seems that the whole interface to the MAD layer may need to be
> reorganized to avoid doing this.

Why not just use inline sends for the special QPs and remove the need to perform
any DMA mappings on the send side altogether?

- Fab