[openib-general] [PATCH] mthca updates (2.6.8 dependent)

[Dror Goldenberg]

> -----Original Message-----
> From: Roland Dreier [mailto:roland at topspin.com] 
> Sent: Saturday, August 14, 2004 7:37 PM

> 
> Now that Linus has released 2.6.8 (and 2.6.8.1), I've just 
> committed a few updates to mthca.  The major change is to use 
> the new MSI/MSI-X API in 2.6.8 (which means that mthca will 
> not build against older kernels).  Since mthca now depends on 
> 2.6.8 anyway, I got rid of the mthca_pci.h file (which just 
> has defines that are all in 2.6.8). Finally, I changed the 
> logic to unconditionally reset the HCA during initialization, 
> since I've found that the previous initialization logic may 
> not work when handed an HCA in an unknown state.

This worth some discussion. What is the case where you expect 
to get the HCA in an unknown state ? Is it when handing over 
between pre-OS driver to the OS-driver ?
It is always good to try as much to quiesce the device before
resetting it. The best way is to SYS_DIS it. Other option for 
a running HCA is to halt if (see CLOSE_HCA with the Panic
close).
Resetting the bus while HCA is running puts some risk if
the PCI-X bus is running, e.g. some PCI reads that got split
response will never get completed.

> 
> To try MSI or MSI-X, you need a platform that supports MSI 
> (right now, that means an Intel system -- I've only tested 
> i386, but x86-64 with an Intel CPU and IA64 should 
> theoretically work too).  Then build your kernel with 
> CONFIG_PCI_MSI=y and add msi=1 and/or msi_x=1 to your mthca 
> module options (if both are set, msi_x will be tried first).
> 
> I believe MSI-X requires HCA firmware that Mellanox has not 
> yet released.  MSI mode seems to work with current firmware 
> (although I have seen some unexplained machine checks when using MSI).

Current Tavor FW is supposed to fully support MSI-X and MSI. You just
choose... I think MSI-X is better :)

> 
> MSI-X seems to improve performance somewhat, because it 
> allows the driver to register three separate interrupt 
> vectors and avoid having to do a slow MMIO read of the event 
> cause register in the interrupt handler.  I don't know if 
> there's much point to plain MSI, because Linux only allows a 
> driver to have a single interrupt even in MSI mode.

I think that limitation of a single vectors is not necessarily Linux
but rather it has to do with the chipset. The major difference between
MSI and MSI-X is where different messages go. While MSI sends
them to the same address, MSI-X sends them to different addresses.
That enables the chipset/CPU-APIC to handle more interrupts messages
per device.
Anyway, I believe that with both MSI and MSI-X you can avoid the 
PIO read. 
* If it's MSI-X, it's easy. By the IRQ# you can tell which EQ has work.
* If it's MSI, just go and peek into all available EQs. It's much more 
   efficient than doing a PIO read.

Last, if you take a look at the verbs extensions. There is a nice 
extension that allows using more than one CQ event handler. The
idea was to use MSI/MSI-X for thing like CPU affinity or IRQ priority.
In which case, you should plan on using multiple MSI/MSI-X vectors
and allow user to select interrupt characteristic. 
... but we're trying to get work done, not talk about verbs extensions,
aren't we :)

> 
> As usual all comments and test reports are gratefully accepted.
> 
>  - Roland
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20040815/b9c5dd72/attachment.html>