[openib-general] IPoIB performance.

[Ido Bukspan]

Hello
I have been investigating the performance of the IPoIB for a while and I
have discovered 3 interesting points
which I think are worth implementing in gen2.

1. We can divide the single CQ into two separate completion queues: one for
the RQ and the other for SQ.
    Then we can change the CQ policy affiliated with the SQ into
IB_CQ_CONSUMER_REARM and in 
    mainstream not arm the CQ.  In such case the poll_cq_tq will be called
from the send_packet method,
    and will reap completions without any need for interrupts/events.
    Obviously in cases when we have to stop the queue ( e.g. no more room
available) , we need 
    to  arm the CQ until completions arrive.
    In general, this change reduces the interrupt rate. It may also help
when posting and polling on the SQ,
    happens from two different processors (e.g. spinlock clash).

2. The current IPoIB driver is signaling every transmitting packet.
    We can improve performance by selective signaling )e.g. every 5 to 10
packets(.
    Note that we did notice several problems when doing it. This approach
can have a problem in case of 
     for example ping (ICMP) which do not allocate new buffers before the
first ones are released. 
     I can think of some WA to this problem such as send a dummy packet
every now or then (which won't go out of the device).
     this includes changing the send policy to be IB_WQ_SIGNAL_SELECTABLE.
     When a selective WQE completes, the ipoib driver has to internally
complete the rest of 
     WQEs that weren't signaled. 
     This change mainly reduces the overhead required by the HW driver to
poll on the completion queue.

3. I think we should call netif_wake_queue only if the queue is actually
stopped because as far as I understand, the kernel
    can schedule another process after calling wake queue.

I have tried these changes and got ~20% improvement in performance )2 Tavor
machines with dual CPU*3.1 GH (Xeon), AS3.0 U3)
If you find it interesting, I can work on a patch for gen2.

What do you think?
-Ido	



Ido Bukspan
Mellanox Technologies Ltd.
Phone : (972)-3-6259500 ,Ext 518.
Fax     : (972)-3-5614943
mailto:ido at mellanox.co.il
http://www.mellanox.com

                                    No play No game