[openib-general] Re: Speeding up IPoIB.

[Grant Grundler]

On Thu, Apr 20, 2006 at 09:03:29PM -0400, Bernard King-Smith wrote:
> Grant> My guess is it's an easier problem to fix SDP than reducing TCP/IP
> Grant> cache/CPU foot print. I realize only a subset of apps can (or will
> Grant> try to) use SDP because of setup/config issues.  I still believe SDP
> Grant> is useful to a majority of apps without having to recompile them.
> 
> I agree that reducing any protocol footprint is a very challenging job,
> however, going to a larger MTU drops the overhead much faster. If IB
> supported a 60K MTU then the TCP/IP overhead would be 1/30 that of what we
> measure today. Traversng the TCP/IP stack once for a 60K packet is much
> lower than 30 times using 2000 byte packets for the same amount of data
> transmitted.

I agree that's effective for workloads which send large messages.
And that's typical for storage workloads.
But the world is not just an NFS server. ;)

> Grant> I'm not competent to disagree in detail.
> Grant> Fabian Tillier and Caitlin Bestler can (and have) addressed this.
> 
> I would be very interested in any pointers to their work.

They have posted to this forum recently on this topic.
The archives are here in case you want to look them up:
	http://www.openib.org/contact.html

> This goes back to systems where the system is busy doing nothing, generally
> when waiting for memory or a cache line miss, or I/O to disks. This is
> where hyperthreading has shown some speedups for benchmarks where
> previously they were totally CPU limited, and with hyperthreading there is
> a gain.

While there are workloads that benefit, I don't buy the hyperthreading
argument in general. Co-workers have demonstrate several "normal"
workloads that don't benefit and are faster with hyperthreading
disabled.

> The unused cycles are "wait" cycles when something can run if it
> can get in quickly. You can't get a TCP stack in the wait, but small parts
> of the stackor driver could fit in the other thread. Yes I do benchmarking
> and was skeptical at first.

ok.

thanks,
grant