[openib-general] [PATCH] 2/2 : ip over ib tx/rx split

[Roland Dreier]

    Michael> I'm reposting this patch, its now tested. Together with
    Michael> the first part: mthca send/receive q lock split, I am
    Michael> getting about 2% more bandwidth with this patch.  I also
    Michael> checked interrupt rate and it is about the same, which is
    Michael> not surprising, since interrupts from both cqs are
    Michael> coalesced by the eq polling in mthca.

Do you see any improvement with just the QP lock split and a single
CQ?  It seems to me that the boost comes from being able to handle
completions from one work queue while posting work requests to the
other queue.  I don't see how having two CQs helps reduce lock
contention or improve parallelism at all.  There's still only one
interrupt handler and one IPoIB completion event handler, which can
only run on one CPU.  So it seems what will happen is:

    EQ event
    Find EQE for receive CQ
    Lock receive CQ, process completions
    Unlock receive CQ
    Find EQE for send CQ
    Lock send CQ, process completions
    Unlock send CQ

The only difference is that we unlock the receive CQ and lock the send
CQ in the middle.

Especially since you say the interrupt rate is the same, it seems that
almost all the time, the completion event handler is finding
completions in both queues anyway.

 - R.