[ofa-general] scp performance over IPoIB

[Rick Jones]

Arlin Davis wrote:
> Rick Jones wrote:
> 
>> Davis, Arlin R wrote:
>>
>>> Can someone explain why scp performance over IPoIB would be 10x slower
>>> then on GBE? The netperf numbers look normal.
>>
>>
>>
>> So, you could try tweaking the MTU on the IPoIB interfaces.
>>
>>
> 
> Rick,
> 
> Thanks for the suggestion. Looks like we may need to change the default 
> MTU for IPoIB. It would be interesting to see results from other 
> distributions.
> 
> (Woodcrest, Xeon 5160, DDR, RHEL4U4)
> 
> MTU      SCP       NetPerf
> 
> 1024    41 MB/s    151 MB/s
> 2048    50 MB/s    313 MB/s
> 4096    50 MB/s    485 MB/s
> 8192    50 MB/s    641 MB/s
> 16384   25 MB/s    761 MB/s
> 32768   50 MB/s    700 MB/s
> 65520   8  MB/s    440 MB/s

I'm actually a triffle surprised that netperf was affected by the 65520 
MTU - I'm guessing you were using all defaults, which on "linux" IIRC 
means netperf was making 16KB (K == 1024) sends.  I suspect that if you 
were to make 64K sends from netperf (test specific -m 64K) that the 
numbers for 64420 might be better.

I'm really shakey on scp behaviour knowledge, but suspect that perhaps 
with the "HPN" (High Performance Network) patches in place (check the 
archives pointed-to by: 
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev) it might be 
possible to get good SCP performance out of a 65520 byte MTU.  I'm 
_guessing" that by default scp isn't trying to put-out > 65520 bytes 
worth of data in the sum of its sends with its own windowing and so gets 
hit by issues with Nagle.  Ie it is doing write, write, read and the 
second write at least is sub-MSS.  Some strace tracing of the scp 
transfer could confirm/deny that hypothesis.

So, it may not be necessary to shrink the MTU.

rick jones