[openib-general] Re: [PATCH] IPoIB splitting CQ, increase both send/recv poll NUM_WC & interval

[Shirley Ma]

Hello Leonid,

Leonid Arsh <leonida at voltaire.com> wrote on 04/23/2006 06:38:00 AM:
> Shirley,
> 
>   some additional information you may be interested:
> 
>   According to our experience with the Voltaire IPoIB driver,
>   splitting CQ harmed the throughput (we checked with the iperf 
> application, UDP mode.) Splitting the the CQ caused more interrupts, 
> context switches and CQ polls.
>   Note, the case is rather different from OpenIB mthca, since Voltare 
> IPoIB is based on the VAPI driver,
>   where CQ completions are handled in a  tasklet context,
>   unlike mthca where  CQ completions are handled in the HW interrupt 
> context.

That expected because only one tasklet is allowed running across all cpus 
in the same time.
Have you tried to use other SOFTIRQ instead of TASKLET_SOFTIRQ?
My expectation is the performance will be better since there would be 
multiple
softirqs running simultaneously. If it's a simple change of your code, 
could you please try it?

I am thinking to split mthca CQ completion into HW interrupt and softirq 
context.

>   NAPI gave us some improvement. I think NAPI should improve much more 
> in mthca, with the HW interrupt CQ completions.
Yes, with the hardware interrupts are disabled.

It would be interesting to compare the completion CQ with NAPI and in 
softirq context.

It all depends on how you implement NAPI. If you only implement NAPI 
without 
changing the sender, NAPI might not get better performance than softirq.
The benefit of NAPI, it has one dev->poll running across all cpus to 
prevent 
packets out of order totally.

Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20060424/63cb9f5c/attachment.html>