<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.45">
<TITLE>RE: [openib-general] [PATCH] mthca updates (2.6.8 dependent)</TITLE>
</HEAD>
<BODY>
<BR>
<BR>
<P><FONT SIZE=2>> -----Original Message-----</FONT>
<BR><FONT SIZE=2>> From: Roland Dreier [<A HREF="mailto:roland@topspin.com">mailto:roland@topspin.com</A>] </FONT>
<BR><FONT SIZE=2>> Sent: Saturday, August 14, 2004 7:37 PM</FONT>
</P>
<P><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Now that Linus has released 2.6.8 (and 2.6.8.1), I've just </FONT>
<BR><FONT SIZE=2>> committed a few updates to mthca. The major change is to use </FONT>
<BR><FONT SIZE=2>> the new MSI/MSI-X API in 2.6.8 (which means that mthca will </FONT>
<BR><FONT SIZE=2>> not build against older kernels). Since mthca now depends on </FONT>
<BR><FONT SIZE=2>> 2.6.8 anyway, I got rid of the mthca_pci.h file (which just </FONT>
<BR><FONT SIZE=2>> has defines that are all in 2.6.8). Finally, I changed the </FONT>
<BR><FONT SIZE=2>> logic to unconditionally reset the HCA during initialization, </FONT>
<BR><FONT SIZE=2>> since I've found that the previous initialization logic may </FONT>
<BR><FONT SIZE=2>> not work when handed an HCA in an unknown state.</FONT>
</P>
<P><FONT SIZE=2>This worth some discussion. What is the case where you expect </FONT>
<BR><FONT SIZE=2>to get the HCA in an unknown state ? Is it when handing over </FONT>
<BR><FONT SIZE=2>between pre-OS driver to the OS-driver ?</FONT>
<BR><FONT SIZE=2>It is always good to try as much to quiesce the device before</FONT>
<BR><FONT SIZE=2>resetting it. The best way is to SYS_DIS it. Other option for </FONT>
<BR><FONT SIZE=2>a running HCA is to halt if (see CLOSE_HCA with the Panic</FONT>
<BR><FONT SIZE=2>close).</FONT>
<BR><FONT SIZE=2>Resetting the bus while HCA is running puts some risk if</FONT>
<BR><FONT SIZE=2>the PCI-X bus is running, e.g. some PCI reads that got split</FONT>
<BR><FONT SIZE=2>response will never get completed.</FONT>
</P>
<P><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> To try MSI or MSI-X, you need a platform that supports MSI </FONT>
<BR><FONT SIZE=2>> (right now, that means an Intel system -- I've only tested </FONT>
<BR><FONT SIZE=2>> i386, but x86-64 with an Intel CPU and IA64 should </FONT>
<BR><FONT SIZE=2>> theoretically work too). Then build your kernel with </FONT>
<BR><FONT SIZE=2>> CONFIG_PCI_MSI=y and add msi=1 and/or msi_x=1 to your mthca </FONT>
<BR><FONT SIZE=2>> module options (if both are set, msi_x will be tried first).</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> I believe MSI-X requires HCA firmware that Mellanox has not </FONT>
<BR><FONT SIZE=2>> yet released. MSI mode seems to work with current firmware </FONT>
<BR><FONT SIZE=2>> (although I have seen some unexplained machine checks when using MSI).</FONT>
</P>
<P><FONT SIZE=2>Current Tavor FW is supposed to fully support MSI-X and MSI. You just</FONT>
<BR><FONT SIZE=2>choose... I think MSI-X is better :)</FONT>
</P>
<P><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> MSI-X seems to improve performance somewhat, because it </FONT>
<BR><FONT SIZE=2>> allows the driver to register three separate interrupt </FONT>
<BR><FONT SIZE=2>> vectors and avoid having to do a slow MMIO read of the event </FONT>
<BR><FONT SIZE=2>> cause register in the interrupt handler. I don't know if </FONT>
<BR><FONT SIZE=2>> there's much point to plain MSI, because Linux only allows a </FONT>
<BR><FONT SIZE=2>> driver to have a single interrupt even in MSI mode.</FONT>
</P>
<P><FONT SIZE=2>I think that limitation of a single vectors is not necessarily Linux</FONT>
<BR><FONT SIZE=2>but rather it has to do with the chipset. The major difference between</FONT>
<BR><FONT SIZE=2>MSI and MSI-X is where different messages go. While MSI sends</FONT>
<BR><FONT SIZE=2>them to the same address, MSI-X sends them to different addresses.</FONT>
<BR><FONT SIZE=2>That enables the chipset/CPU-APIC to handle more interrupts messages</FONT>
<BR><FONT SIZE=2>per device.</FONT>
<BR><FONT SIZE=2>Anyway, I believe that with both MSI and MSI-X you can avoid the </FONT>
<BR><FONT SIZE=2>PIO read. </FONT>
<BR><FONT SIZE=2>* If it's MSI-X, it's easy. By the IRQ# you can tell which EQ has work.</FONT>
<BR><FONT SIZE=2>* If it's MSI, just go and peek into all available EQs. It's much more </FONT>
<BR><FONT SIZE=2> efficient than doing a PIO read.</FONT>
</P>
<P><FONT SIZE=2>Last, if you take a look at the verbs extensions. There is a nice </FONT>
<BR><FONT SIZE=2>extension that allows using more than one CQ event handler. The</FONT>
<BR><FONT SIZE=2>idea was to use MSI/MSI-X for thing like CPU affinity or IRQ priority.</FONT>
<BR><FONT SIZE=2>In which case, you should plan on using multiple MSI/MSI-X vectors</FONT>
<BR><FONT SIZE=2>and allow user to select interrupt characteristic. </FONT>
<BR><FONT SIZE=2>... but we're trying to get work done, not talk about verbs extensions,</FONT>
<BR><FONT SIZE=2>aren't we :)</FONT>
</P>
<P><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> As usual all comments and test reports are gratefully accepted.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> - Roland</FONT>
</P>
</BODY>
</HTML>