[openib-general] How about ib_send_page() ?

[Jeff Carr]

Grant Grundler wrote:

> If it's NAPI that means nothing, here's probably the best summary:
> 	http://lwn.net/Articles/30098/

Cool; I see now.

>>>But IPoIB can't really implement NAPI since it's sending work to
>>>a shared HCA. 

Hmm. I'm not knowledgeable to know why; I'll have to take your word for 
it. I'm not sure yet all the conditions that the HCA can generate 
interrupts.

But if I sit back and look at the logic of this arguement then it seemed 
like:

Hey. is there was a way to not generate so many interrupts?
That's handled by NAPI
OK. That looks interesting.
But, we can't do NAPI because we can't just disable interrupts.
Darn.
But wait, why can't we just not generate interrupts in the first place then?

Isn't that what the midas touch of netdev->poll() really is? e1000 has:
quit_polling:   netif_rx_complete(netdev);
                 e1000_irq_enable(adapter);

Maybe IB can mimic the concept here by acting intellegently for us? 
Have disable_rx_and_rxnobuff_ints() only disable interrupts for the 
IPoIB ULP? Anyway, my knowledge here still sucks so I'm probably so far 
off base I'm not even on the field. Either way it's fun digging around here.

> NAPI, TOE, and lots of other things help ethernet with throughput.

Hmmm, TOE is interesting.

> One can.  Using SDP, netperf TCP_STREAM measured 650 MB/s using the
> regular PCI-X card. 

Yes, I have the same speed results using perf_main().

The perf_main() test isn't that interesting I think though. It really 
just transfers the exact same memory window across two nodes. (at least 
as far as I can tell that is what it does).

Anyway, I'm just noticing that this simple dd test from memory doesn't 
go much over 1GB/sec. So this is an interesting non-IB problem.

root at jcarr:/# dd if=/dev/shm/test of=/dev/null bs=4K
196608+0 records in
196608+0 records out
805306368 bytes transferred in 0.628504 seconds (1281306571 bytes/sec)

> I wonder if that could be faster with bigger socket
> buffers but don't know (IIRC, IB socket buffers are fixed at 256KB).
> Anyway, that's with about 60% CPU (1.5 Ghz ia64).

I've got a 3.2Ghz Xeon. 2GB ram.

> I haven't checked to see what libibverbs can do with "native" mode
> (if one is willing/able to modify the application.)

42

> ISTR Michael Tsirkin had another streaming test in mind to measure
> peak bandwidth. But I'm not quite done with rdma_lat yet anyway...

Enjoy,
Jeff