[openib-general] [PATCH] splitting IPoIB CQ

[Roland Dreier]

 > > Actually, do you have some explanation for why this helps performance?
 > > My intuition would be that it just generates more interrupts for the
 > > same workload.

 > The only lock contension in IPoIB I saw is tx_lock. When seperating 
 > the completion queue to have seperate completion handler. It could improve
 > the performance. I didn't look at driver code, it might have some impact 
 > there? 

A clever way to avoid taking the TX lock on send completions would be
very nice, but I never saw a way to do it.  Does splitting the CQ
reduce contention?  I don't see why that would be, since the
contention is between sending and getting send completions.  The
receive path of course never touches the tx_lock.

I'm a little uncomfortable splitting the IPoIB CQ until we have a
plausible theory as to why it improves performance.  It may be that
understanding this would lead us to fix the underlying issue in a
better way and get even more performance.

By the way, are your numbers with mthca or ehca?  I don't know ehca
very well, but at least with current mthca, all CQ events will be
delivered on the same interrupt and hence all CQ handling will run on
the same CPU.  So I'm puzzled why changing things from:

    -> interrupt
    -> CQ event handler
    -> handle all IPoIB completions

to:

    -> interrupt
    -> TX CQ event handler
    -> handle TX completions
    [possibly another interrupt]
    -> RX CQ event handler
    -> handle RX completions

helps throughput.  It just seems like it's more CQ locking/unlocking
and in general more work.

 - R.