<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=US-ASCII">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.45">
<TITLE>RE: [openib-general] [PATCH] mthca updates (2.6.8 dependent)</TITLE>
</HEAD>
<BODY>
<BR>
<BR>
<P><FONT SIZE=2>> -----Original Message-----</FONT>
<BR><FONT SIZE=2>> From: Grant Grundler [<A HREF="mailto:iod00d@hp.com">mailto:iod00d@hp.com</A>] </FONT>
<BR><FONT SIZE=2>> Sent: Tuesday, August 31, 2004 8:23 AM</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> On Mon, Aug 30, 2004 at 09:10:35PM -0700, Roland Dreier wrote:</FONT>
<BR><FONT SIZE=2>> > The Intel E7500 Xeon chipset PCI bridge datasheet does say that an </FONT>
<BR><FONT SIZE=2>> > interrupt message causes the bridge to flush its write buffers to </FONT>
<BR><FONT SIZE=2>> > preserve precisely this ordering, but I don't know whether everyone </FONT>
<BR><FONT SIZE=2>> > followed or will follow this example (I'm sure we'll see </FONT>
<BR><FONT SIZE=2>> many more MSI </FONT>
<BR><FONT SIZE=2>> > implementations on PCI Express).</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Maybe the flush is required because of DMA write coalescing </FONT>
<BR><FONT SIZE=2>> in the E7500 chipset?</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Ie any DMA writes which can't be coalesced will cause this </FONT>
<BR><FONT SIZE=2>> kind of a flushing behavior. I don't really know since I </FONT>
<BR><FONT SIZE=2>> don't how E7500 handles cache coherency.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> And yes, I'm certain some chipsets get DMA write coalescing </FONT>
<BR><FONT SIZE=2>> wrong. Look at drivers/net/tg3.c and search for </FONT>
<BR><FONT SIZE=2>> TG3_FLAG_MBOX_WRITE_REORDER in tg3_get_invariants().</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> grant</FONT>
</P>
<P><FONT SIZE=2>I think that one of the most important intentions of the MSI was to</FONT>
<BR><FONT SIZE=2>save the need to perform the MMIO read which is expensive in CPU</FONT>
<BR><FONT SIZE=2>time. I think that for mainstream systems you can assume that there</FONT>
<BR><FONT SIZE=2>is ordering between MSI and other DMA writes. If some systems</FONT>
<BR><FONT SIZE=2>have related bugs, then in this case they can add a workaround (which</FONT>
<BR><FONT SIZE=2>may be either to perform a MMIO read, or to use good old regular</FONT>
<BR><FONT SIZE=2>interrupts.</FONT>
<BR><FONT SIZE=2>Another thing worth mentioning, is that the InfiniHost will fire an interrupt</FONT>
<BR><FONT SIZE=2>again if the interrupt handler didn't clean up the event queue. I.e. in the </FONT>
<BR><FONT SIZE=2>rare (or non existing case) of the MSI bypassing the preceding DMA </FONT>
<BR><FONT SIZE=2>write, if the driver doesn't see the EQE in memory, at the time it rearms</FONT>
<BR><FONT SIZE=2>the EQ, it'll get an immediate interrupt. So, you should be safe anyway...</FONT>
</P>
<P><FONT SIZE=2>BTW, here's an excerpt from the PCI spec* explaining the the driver need</FONT>
<BR><FONT SIZE=2>not do any MMIO read from the device in order to ensure that the data is </FONT>
<BR><FONT SIZE=2>in memory at the time the MSI is received:</FONT>
</P>
<P><FONT SIZE=2>"An MSI or MSI-X message, by virtue of being a posted memory write (PMW) transaction, is prohibited by PCI ordering rules from passing PMW transactions sent earlier by the function. The system must guarantee that an interrupt service routine invoked as a result of a given message will observe any updates performed by PMW transactions arriving prior to that message. Thus, the interrupt service routine of a device driver is not required to read from a device register in order to ensure data consistency with previous PMW transactions. However, if multiple MSI-X Table entries share the same vector, the interrupt service routine may need to read from some device specific register to determine which interrupt sources need servicing."</FONT></P>
<P><FONT SIZE=2>* Section 6.8.3.6 at the end in <A HREF="http://www.pcisig.com/specifications/conventional/msi-x_ecn.pdf" TARGET="_blank">http://www.pcisig.com/specifications/conventional/msi-x_ecn.pdf</A></FONT>
</P>
<P><FONT SIZE=2>-Dror</FONT>
</P>
</BODY>
</HTML>