[openib-general] Re: Re: [PATCH] IPoIB splitting CQ, increase both send/recv poll NUM_WC & interva

[Bernard King-Smith]

Michael S. Tsirkin wrote:

Michael> Quoting r. Shirley Ma <xma at us.ibm.com>:
Michael> > different drivers have different implementations for CQ
completion handler.

Michael> Maybe these drivers should be changed then?  Its a bit hard for me
to imagine a
Michael> driver that doesn't get hardware interrupts in IRQ context. So why
can't
Michael> completion handler be called directly from there as well?

The problem we are seeing is not how we handle the hardware interrupt, but
what is handled where, after getting the interrupt. All packet processing
is handled by a single processor. On a multiprocessor machine with high
traffic rate, the bandwidth is limited by a single pro, and to different
CPU's than the one handling the interrupt. Running the 10 GigE adapter in
classic packet mode ( 1500 byte MTU ) gets better performance with
splitting the packet handling across multiple CPU's. IB is suffering from
this same problem in the completion handler. If you have a 4 CPU machine
and run two sockets over teh same adapter/link using NETPERF, if you use
mpstat -P ALL, you will see one CPU pegged and the rest much lower in CPU
utilization, The CPU at 100% is the one running the interrupt handler,
TCP/IP stack and completion queue.

Shirley's patch is to split the processing so that completion queue
handling can be done on a different CPU that the interrupt handler. Yes
there are locking issues with the completion queue being accessed by
different CPU's but there is an overall gain in bandwidth especially in
duplex mode. Having all completion queue handling done off the hardware
interrupt means you have serialized the completions for a duplex adapter.

Michael> --
Michael> MST

Bernie King-Smith
IBM Corporation
Server Group
Cluster System Performance
wombat2 at us.ibm.com    (845)433-8483
Tie. 293-8483 or wombat2 on NOTES

"We are not responsible for the world we are born into, only for the world
we leave when we die.
So we have to accept what has gone before us and work to change the only
thing we can,
-- The Future." William Shatner