[openib-general] How about ib_send_page() ?

[Grant Grundler]

On Mon, May 16, 2005 at 06:00:49PM -0700, Jeff Carr wrote:
> with vmstat on the server showing:
> 
> -----memory---------- ---swap-- -----io---- --system-- ----cpu----
>  swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
>   0 1115720 440988  49836    0    0     0     0 43943  8251  0 94  6  0
>   0 1115720 440988  49836    0    0     0     0 43725  8384  0 94  6  0

vmstat doesn't tell you where the time is being spent.
Get a profile or try out the beta Pentium M or AMD64 perfmon support
that was recently posted:
	http://www.hpl.hp.com/hosted/linux/mail-archives/perfmon/2005-May/000389.html

perfmon homepage doesn't reflect the beta version but gives
a good overview:
	http://www.hpl.hp.com/research/linux/perfmon/

You also want to explore "netperf -C" option that' available with
netperf 2.4.0-rc1 (See www.netperf.orf).  I've posted results here
before about binding netperf/netserver processes to different CPUs.

> So the CPU is throttled at around 100-200k packets/sec with 43k
> interrupts / sec.  Perhaps the HCA can be configured to respond/trigger
> events/generate interrupts less often under higher load. Some ethernet
> chips have this kind of functionality (interrupt after x packets or
> after n time).

NAPI obsoletes interrupt coalescing to a large degree on regular NICs.
But IPoIB can't really implement NAPI since it's sending work to
a shared HCA. And any form of interrupt coalescing would interfere with
any latency sensitive work as well (if present).

grant