[openib-general] Race in mthca_cmd_post()

[Jason Gunthorpe]

On Mon, Oct 23, 2006 at 08:00:45PM -0700, Roland Dreier wrote:
>  > I just took a quick look at asm-ia64/io.h and there is __ia64_mf_a
>  > barriers after all non-posted IO operations (ib/outb). config write and
>  > config read transcations have identical rules to IO transactions at
>  > the PCI bus level.
>  > 
>  > I'm going to go out on a limb here and say that if Linux code assumes
>  > strong ordering of IO operations then it makes sense to also assume
>  > strong ordering on config writes. So, instead of patching mthca with
>  > this barrier it should go in the Altix config access mechanism..
 
> I don't really know what mf.a does on ia64, but it seems likely that
> even if the CPU issues and retires the reads and writes in order
> (which is all a CPU barrier is likely to do), we would still have a
> problem where the PCI-X host bridge allows the MMIO read to pass the
> config write, because the config write is still pending a split
> completion on the bus.

Well, I'm in the same boat as you as far as Altix goes, but in
general, on multi-processor systems some barrier instructions can
produce fencing operations on the CPU bus that a PCI-X bridge or
chipset might observe. Eventually in that discussion thread you found
there is this:

  > Well, mf.a *does* do something on the 460 chipset.  So I think it's
  > mostly a platform issue.  Are you saying that on SGI's IA-64 platforms
  > mf.a doesn't do the equivalent of the MIPS sync?  (Just curious.)

So on some Itaniums mf.a comes out the CPU bus in some manner..

After thinking about it some more, there are more cases in the kernel
than just the one in mthca. For instance there are paths in the pci
core than manipulate BAR registers without a barrier. It would be bad
to change a bar via config write and issue a MMIO operation to the new
address without waiting for the split to return. I also know of a
fibre channel chip that has a similar strong requirement of order in
the reset sequence like mthca.

Regards,
Jason