[ofa-general] synchronize commands issued to MTHCA

[Roland Dreier]

 > Could you tell me what's the difference between "wmb()" and "mmiowb()". I 
 > notice that ofa-1.3 has added "mmiowb()" at the end of mthca_cmd_post, 
 > since "wmb()" is already called at the end of cmd_post, is "mmiowb()" 
 > really necessary?

wmb() orders writes from the same CPU -- it prevents highly
out-of-order architectures from making writes visible in an order
different from program order.  mmiowb() orders MMIO writes between
different CPUs, and prevents systems (such as SGI Altix) where the CPU
fabric may reorder writes before they reach the IO bus.

The mmiowb() is definitely necessary, because without it then commands
were getting messed up on large Altix systems.

 - R.