[openib-general] Re: Speeding up IPoIB.

[Bernard King-Smith]

Grant Grundler wrote:
> Grant> My guess is it's an easier problem to fix SDP than reducing TCP/IP
> Grant> cache/CPU foot print. I realize only a subset of apps can (or will
> Grant> try to) use SDP because of setup/config issues.  I still believe
SDP
> Grant> is useful to a majority of apps without having to recompile them.
>
> I agree that reducing any protocol footprint is a very challenging job,
> however, going to a larger MTU drops the overhead much faster. If IB
> supported a 60K MTU then the TCP/IP overhead would be 1/30 that of what
we
> measure today. Traversng the TCP/IP stack once for a 60K packet is much
> lower than 30 times using 2000 byte packets for the same amount of data
> transmitted.

Grant> I agree that's effective for workloads which send large messages.
Grant> And that's typical for storage workloads.
Grant> But the world is not just an NFS server. ;)

However, NFS is not the only large data transfer workload I come across. If
IB wants to achieve high volumes there needs to be some kind of commercial
workload that works well on it besides the large applications that can
afford to port to SDP or uDAPL. HPC is a nitch that is hard to sustain a
viable business in ( though everyone can point to a couple of companies,
most are not long term, 15 years or more ).

However, in clustering the areas that benefit from large block transfer
efficiency include:

File Serving ( NFS, GPFS, XFS etc.)
Application backup
Parallel databases
Database upload/update flows
Web server graphics
Web server MP3s
Web server streaming video
Local workstation backup
Collaboration software
Local mail replication

My concern is that if IB does not support these operations as well as
Ethernet, then it is a hard sell into commercial accounts/workloads for IB.

Bernie King-Smith
IBM Corporation
Server Group
Cluster System Performance
wombat2 at us.ibm.com    (845)433-8483
Tie. 293-8483 or wombat2 on NOTES

"We are not responsible for the world we are born into, only for the world
we leave when we die.
So we have to accept what has gone before us and work to change the only
thing we can,
-- The Future." William Shatner


                                                                           
             Grant Grundler                                                
             <iod00d at hp.com>                                               
                                                                        To 
             04/21/2006 04:10          Bernard                             
             PM                        King-Smith/Poughkeepsie/IBM at IBMUS   
                                                                        cc 
                                       Grant Grundler <iod00d at hp.com>,     
                                       openib-general at openib.org, Roland   
                                       Dreier <rdreier at cisco.com>          
                                                                   Subject 
                                       Re: Speeding up IPoIB.              
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           




On Thu, Apr 20, 2006 at 09:03:29PM -0400, Bernard King-Smith wrote:
> Grant> My guess is it's an easier problem to fix SDP than reducing TCP/IP
> Grant> cache/CPU foot print. I realize only a subset of apps can (or will
> Grant> try to) use SDP because of setup/config issues.  I still believe
SDP
> Grant> is useful to a majority of apps without having to recompile them.
>
> I agree that reducing any protocol footprint is a very challenging job,
> however, going to a larger MTU drops the overhead much faster. If IB
> supported a 60K MTU then the TCP/IP overhead would be 1/30 that of what
we
> measure today. Traversng the TCP/IP stack once for a 60K packet is much
> lower than 30 times using 2000 byte packets for the same amount of data
> transmitted.

I agree that's effective for workloads which send large messages.
And that's typical for storage workloads.
But the world is not just an NFS server. ;)

> Grant> I'm not competent to disagree in detail.
> Grant> Fabian Tillier and Caitlin Bestler can (and have) addressed this.
>
> I would be very interested in any pointers to their work.

They have posted to this forum recently on this topic.
The archives are here in case you want to look them up:
             http://www.openib.org/contact.html

> This goes back to systems where the system is busy doing nothing,
generally
> when waiting for memory or a cache line miss, or I/O to disks. This is
> where hyperthreading has shown some speedups for benchmarks where
> previously they were totally CPU limited, and with hyperthreading there
is
> a gain.

While there are workloads that benefit, I don't buy the hyperthreading
argument in general. Co-workers have demonstrate several "normal"
workloads that don't benefit and are faster with hyperthreading
disabled.

> The unused cycles are "wait" cycles when something can run if it
> can get in quickly. You can't get a TCP stack in the wait, but small
parts
> of the stackor driver could fit in the other thread. Yes I do
benchmarking
> and was skeptical at first.

ok.

thanks,
grant