[openib-general] [PATCH] mthca updates (2.6.8 dependent)

[Roland Dreier]

    Roland> Finally, I changed the logic to unconditionally reset the
    Roland> HCA during initialization, since I've found that the
    Roland> previous initialization logic may not work when handed an
    Roland> HCA in an unknown state.

    Dror> This worth some discussion. What is the case where you
    Dror> expect to get the HCA in an unknown state ? Is it when
    Dror> handing over between pre-OS driver to the OS-driver ?  It is
    Dror> always good to try as much to quiesce the device before
    Dror> resetting it. The best way is to SYS_DIS it. Other option
    Dror> for a running HCA is to halt if (see CLOSE_HCA with the
    Dror> Panic close).  Resetting the bus while HCA is running puts
    Dror> some risk if the PCI-X bus is running, e.g. some PCI reads
    Dror> that got split response will never get completed.

I'll check into this.  My impression is that we had problems running
FW commands when the HCA was left running by a pre-OS driver.

    Roland> MSI-X seems to improve performance somewhat, because it
    Roland> allows the driver to register three separate interrupt
    Roland> vectors and avoid having to do a slow MMIO read of the
    Roland> event cause register in the interrupt handler.  I don't
    Roland> know if there's much point to plain MSI, because Linux
    Roland> only allows a driver to have a single interrupt even in
    Roland> MSI mode.

    Dror> I think that limitation of a single vectors is not
    Dror> necessarily Linux but rather it has to do with the
    Dror> chipset. The major difference between MSI and MSI-X is where
    Dror> different messages go. While MSI sends them to the same
    Dror> address, MSI-X sends them to different addresses.  That
    Dror> enables the chipset/CPU-APIC to handle more interrupts
    Dror> messages per device.

Maybe, but the Linux Documentation/MSI-HOWTO.txt file says this about
MSI vectors:

    "Due to the non-contiguous fashion in vector assignment of the
    existing Linux kernel, this version does not support multiple
    messages regardless of a device function is capable of supporting
    more than one vector."

    Dror> Anyway, I believe that with both MSI and MSI-X you can avoid
    Dror> the PIO read. * If it's MSI-X, it's easy. By the IRQ# you
    Dror> can tell which EQ has work.  * If it's MSI, just go and peek
    Dror> into all available EQs. It's much more efficient than doing
    Dror> a PIO read.

It seems this would apply to standard INTx mode as well.  Do you know
why Mellanox didn't use this in THCA?  In any case, I'll have to
benchmark this approach.

    Dror> Last, if you take a look at the verbs extensions. There is a
    Dror> nice extension that allows using more than one CQ event
    Dror> handler. The idea was to use MSI/MSI-X for thing like CPU
    Dror> affinity or IRQ priority.  In which case, you should plan on
    Dror> using multiple MSI/MSI-X vectors and allow user to select
    Dror> interrupt characteristic. ... but we're trying to get work
    Dror> done, not talk about verbs extensions, aren't we :)

Yes, I've definitely thought of this, and it's a great idea.  However,
I thought it would be worth it to get our base verbs working before
trying to design an API for multiple CQ event handlers.  It's definitely
something I'll implement in the future (the actual mthca code is
pretty easy; designing the access layer API is most of the work).

Thanks,
 - R.