[ofa-general] scp performance over IPoIB

[Davis, Arlin R]

Can someone explain why scp performance over IPoIB would be 10x slower
then on GBE? The netperf numbers look normal.

Running OFED 1.2, IPoIB-cm

[ardavis at C-27-61 ~]$ /sbin/ifconfig ib0
ib0       Link encap:UNSPEC  HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00          inet
addr:36.102.27.61  Bcast:36.102.255.255  Mask:255.255.0.0
         UP BROADCAST RUNNING MULTICAST  MTU:65520  Metric:1
         RX packets:399017 errors:0 dropped:0 overruns:0 frame:0
         TX packets:509466 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:128
         RX bytes:735453689 (701.3 MiB)  TX bytes:11065632473 (10.3 GiB)


scp performance:

ethernet:
[ardavis at C-27-61 ~]$ scp 36.101.27.60:/tmp/testfile3 /tmp/testfile3

testfile3
100%  215MB  53.7MB/s   00:04   

infiniband:
[ardavis at C-27-61 ~]$ scp 36.102.27.60:/tmp/testfile3 /tmp/testfile3
testfile3
100%  215MB   5.8MB/s   00:37   

netperf performance:

ethernet:

[ardavis at C-27-61 ~]$  netperf -f -M -c -C -H 36.101.27.60
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 36.101.27.60
(36.101.27.60) port 0 AF_INET
Recv   Send    Send                          Utilization       Service
Demand
Socket Socket  Message  Elapsed              Send     Recv     Send
Recv
Size   Size    Size     Time     Throughput  local    remote   local
remote
bytes  bytes   bytes    secs.    KBytes  /s  % S      % S      us/KB
us/KB

87380  16384  16384    10.00      114501.08   6.59     11.74    2.302
4.100

infiniband:

[ardavis at C-27-61 ~]$  netperf -f -M -c -C -H 36.102.27.60
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 36.102.27.60
(36.102.27.60) port 0 AF_INET
Recv   Send    Send                          Utilization       Service
Demand
Socket Socket  Message  Elapsed              Send     Recv     Send
Recv
Size   Size    Size     Time     Throughput  local    remote   local
remote
bytes  bytes   bytes    secs.    KBytes  /s  % S      % S      us/KB
us/KB

87380  16384  16384    10.00      340012.36   15.12    16.59    1.778
1.952


Thanks,

-arlin