[openib-general] How about ib_send_page() ?

[Grant Grundler]

On Mon, May 16, 2005 at 07:20:07PM -0700, Jeff Carr wrote:
> >You also want to explore "netperf -C" option that' available with
> >netperf 2.4.0-rc1 (See www.netperf.orf).  I've posted results here
> >before about binding netperf/netserver processes to different CPUs.
> 
> I think I remember this thread. I'll look back for it again.
> 
> It does look like the -C option is in 2.3.4 though:

Sorry - I meant "-T".
And that seems to have been added to 2.3pl2 as well.

> >NAPI obsoletes interrupt coalescing to a large degree on regular NICs.
> 
> ?

42. :^)

If it's NAPI that means nothing, here's probably the best summary:
	http://lwn.net/Articles/30098/

> 
> >But IPoIB can't really implement NAPI since it's sending work to
> >a shared HCA. 
> 
> > And any form of interrupt coalescing would interfere with
> >any latency sensitive work as well (if present).
> 
> Surely. It would have to be configureable for people (like me) that 
> wanted it. I don't care about latency, just about throughput.

NAPI, TOE, and lots of other things help ethernet with throughput.
None of those help with latency and one has to spend alot of time
tuning to get things right, much less as a tradeoff.
I'm blissfully ignorant of any tuning that goes on for openib.org.

> What's the 
> point of having a fast network if you can't send *lots* of data over it. :)

One can.  Using SDP, netperf TCP_STREAM measured 650 MB/s using the
regular PCI-X card. I wonder if that could be faster with bigger socket
buffers but don't know (IIRC, IB socket buffers are fixed at 256KB).
Anyway, that's with about 60% CPU (1.5 Ghz ia64).
I haven't checked to see what libibverbs can do with "native" mode
(if one is willing/able to modify the application.)

ISTR Michael Tsirkin had another streaming test in mind to measure
peak bandwidth. But I'm not quite done with rdma_lat yet anyway...

grant