[ofa-general] scp performance over IPoIB

[Rick Jones]

Davis, Arlin R wrote:
> Can someone explain why scp performance over IPoIB would be 10x slower
> then on GBE? The netperf numbers look normal.

Might you be running into limitations of the app-level windowing in scp 
(ssl?).  ISTR there is need of patching to get scp to work "well" with 
high bandwidth delay product links.

If your IPoIB happens to have higher latency than your GbE setup - which 
you can check with a netperf TCP_RR test then perhaps the Tput < W/RTT 
thing is happening - with the app-level windowing.

Might also check the send sizes in scp relative to the MTU - the OFED 
1.2 IPoIB MTU is (IIRC) 65520 bytes and perhaps the scp sends aren't 
playing well with that.  Netperf TCP_STREAM will just shove bytes into 
the socket until blocked, which scp may not do.

So, you could try tweaking the MTU on the IPoIB interfaces.

rick jones