[openib-general] Re: [PATCH] IPoIB splitting CQ, increase both send/recv poll NUM_WC & interval

[Leonid Arsh]

Please correct me if I'm mistaken.
I think that soft IRQ is used by the kernel for handling the received 
packets - NET_RX_SOFTIRQ.

For the non-NAPI case, we poll the CQ for completions and call 
netif_rx() in the completion notification context -
the HW interrupt context in case of mthca.
netif_rx()  queues the received skb and schedules  the  NET_RX_SOFTIRQ 
by calling netif_rx_schedule().
In the same context   the driver requests the HCA to notify new completions.
The skb will be passed to the TCP/IP stack later in the soft IRQ context 
by calling netif_receive_skb()
(see the process_backlog() kernel function)

For the NAPI case,  the driver shall only call netif_rx_schedule() in 
the completion notification context -
the HW interrupt context in case of mthca.
The CQ polling and passing the skb to the TCP/IP stack will be done in 
the NET_RX_SOFTIRQ context in the driver's poll() function. Only after 
that the driver will request the HCA to notify new completions.

The late completion notification request should cause the interrupt 
mitigation -- and this is the main NAPI advantage, for my opinion.

Of course, the NAPI implementation may need the CQ splitting.
On the other hand, polling CQ for the send and receive completions in 
the same context my be an advantage for
bidirectional traffic.

BTW, the HW interrupt coalescing used to help us in some cases.

Regards,
   Leonid


Shirley Ma wrote:
>
> Hello Leonid,
>
> Leonid Arsh <leonida at voltaire.com> wrote on 04/23/2006 06:38:00 AM:
> > Shirley,
> >
> >   some additional information you may be interested:
> >
> >   According to our experience with the Voltaire IPoIB driver,
> >   splitting CQ harmed the throughput (we checked with the iperf
> > application, UDP mode.) Splitting the the CQ caused more interrupts,
> > context switches and CQ polls.
> >   Note, the case is rather different from OpenIB mthca, since Voltare
> > IPoIB is based on the VAPI driver,
> >   where CQ completions are handled in a  tasklet context,
> >   unlike mthca where  CQ completions are handled in the HW interrupt
> > context.
>
> That expected because only one tasklet is allowed running across all 
> cpus in the same time.
> Have you tried to use other SOFTIRQ instead of TASKLET_SOFTIRQ?
> My expectation is the performance will be better since there would be 
> multiple
> softirqs running simultaneously. If it's a simple change of your code, 
> could you please try it?
>
> I am thinking to split mthca CQ completion into HW interrupt and 
> softirq context.
>
> >   NAPI gave us some improvement. I think NAPI should improve much more
> > in mthca, with the HW interrupt CQ completions.
> Yes, with the hardware interrupts are disabled.Re: [PATCH] splitting 
> IPoIB CQ 
> <http://news.gmane.org/find-root.php?message_id=%3cOFCC3068D1.F31E1D05%2dON87257155.00606301%2d88257155.0065DBBF%40us.ibm.com%3e>
>
> It would be interesting to compare the completion CQ with NAPI and in 
> softirq context.
>
> It all depends on how you implement NAPI. If you only implement NAPI 
> without
> changing the sender, NAPI might not get better performance than softirq.
> The benefit of NAPI, it has one dev->poll running across all cpus to 
> prevent
> packets out of order totally.
>
> ThanksRe: [PATCH] splitting IPoIB CQ 
> <http://news.gmane.org/find-root.php?message_id=%3cOFCC3068D1.F31E1D05%2dON87257155.00606301%2d88257155.0065DBBF%40us.ibm.com%3e>
> Shirley Ma
> IBM Linux Technology Center
> 15300 SW Koll Parkway
> Beaverton, OR 97006-6063
> Phone(Fax): (503) 578-7638