[ofa-general] Re: should it be possible to run SDP over a T320?

[Rick Jones]

> I am only interested in IPoIB at this moment for both ofed-1.1 and 
> ofed-1.2. Is the device PCI-X or PCI-e based?

Well, I guess that's better than "everything" :)  but it is still a triffle 
broad.  Anyway, I'll suppress my "sending to another .com" paranoia by remiding 
myself that all this is shipping :) and include the results here.

The device is PCIe.  lspci shows:

03:00.0 InfiniBand: Mellanox Technologies MT25208 InfiniHost III Ex (Tavor 
compatibility mode) (rev 20)

RHEL5 included OFED 1.1 bits:

rx2660 to rx2660, rhel5, AD313A, IPoIB and now SDP, same
sysctl settings, irqbalance killed to keep things from moving around,
interrups on the HCA now on cpu0 on one system and cpu 1 on the other

[here are the sysctl.conf settings:

[root at hpcpc107 ~]# tail /etc/sysctl.conf
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 2147483648


net.core.rmem_max = 2097152
net.core.wmem_max = 2097152
net.ipv4.tcp_wmem = 4096 87380 2097152
net.ipv4.tcp_rmem = 4096 87380 2097152
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_filter = 1

]

[ the first number is the CPU to which netperf is bound, the second is the CPU 
to which netserver is bound.  the systems under test had _four_ cores, which 
means that when netperf reports 25% CPU util it means the equivalent of a full 
core was consumed etc etc ]

single-connection, unidirectional TCP_STREAM 1Mx64:
[root at hpcpc106 netperf2_work]# for i in 0 1; do for j in 0 1; do echo $i $j 
`netperf -P 0 -T $i,$j -c -C -H 192.168.0.107 -t TCP_STREAM -i 30,3 -l 30 -- -s 
1M -S 1M -m 64K`; done; done
0 0 2097152 2097152 65536 30.00 2382.94 25.00 37.29 3.438 5.128
0 1 2097152 2097152 65536 30.00 2315.88 19.97 25.03 2.826 3.542
1 0 2097152 2097152 65536 30.00 2974.46 40.10 41.25 4.418 4.544 *
1 1 2097152 2097152 65536 30.00 2358.11 27.39 25.01 3.807 3.476


[NOTE NOTE NOTE - the units here are still transactions per second! so, to get 
to mbit/s multiply by 2x65536x8 and divide by 1000000...  To get the service 
demand in usec per KB transferred, divide the service demand by 128 since that 
was the number of KB transferred per transaction]

single-connection, bidrectional TCP_RR 1Mx64x, ad313a hca in x8 slot
[root at hpcpc106 netperf2_work]# for i in 0 1 ; do for j in 0 1 ; do echo $i $j 
`netperf -P 0 -T $i,$j -c -C -H 192.168.0.107 -t TCP_RR -i 30,3 -l 30 -- -s 1M 
-S 1M -r 64K -b 12`; done; done
0 0 2097152 2097152 65536 65536 30.00 2485.29 25.01 31.45 402.595 506.196 
2097152 2097152
0 1 2097152 2097152 65536 65536 30.00 2414.18 23.38 23.33 387.354 386.507 
2097152 2097152
1 0 2097152 2097152 65536 65536 30.00 3368.20 38.75 38.72 460.153 459.788 
2097152 2097152 *
1 1 2097152 2097152 65536 65536 30.00 2504.03 31.54 25.05 503.753 400.236 
2097152 2097152

[NOTE NOTE NOTE - when netperf reports a confidence of 20.7% it means +/- 10.35%]

single-connection, single-byte, TCP_RR, ad313a hca in x8 slot:
[root at hpcpc106 netperf2_work]# for i in 0 1 ; do for j in 0 1 ; do echo $i $j 
`netperf -P 0 -T $i,$j -c -C -H 192.168.0.107 -t TCP_RR -i 30,3 -l 30 `; done; done
0 0 !!! WARNING !!! Desired confidence was not achieved within the specified 
iterations. !!! This implies that there was variability in the test environment 
that !!! must be investigated before going further. !!! Confidence intervals: 
Throughput : 0.1% !!! Local CPU util : 20.7% !!! Remote CPU util : 13.7% 87380 
87380 1 1 30.00 15743.40 4.84 10.08 12.293 25.610 87380 87380
0 1 !!! WARNING !!! Desired confidence was not achieved within the specified 
iterations. !!! This implies that there was variability in the test environment 
that !!! must be investigated before going further. !!! Confidence intervals: 
Throughput : 0.4% !!! Local CPU util : 59.3% !!! Remote CPU util : 51.1% 87380 
87380 1 1 30.00 19298.77 4.70 7.09 9.751 14.694 87380 87380
1 0 !!! WARNING !!! Desired confidence was not achieved within the specified 
iterations. !!! This implies that there was variability in the test environment 
that !!! must be investigated before going further. !!! Confidence intervals: 
Throughput : 0.2% !!! Local CPU util : 28.6% !!! Remote CPU util : 34.4% 87380 
87380 1 1 30.00 13016.11 6.15 6.57 18.912 20.195 87380 87380
1 1 !!! WARNING !!! Desired confidence was not achieved within the specified 
iterations. !!! This implies that there was variability in the test environment 
that !!! must be investigated before going further. !!! Confidence intervals: 
Throughput : 0.1% !!! Local CPU util : 8.7% !!! Remote CPU util : 23.4% 87380 
87380 1 1 30.00 15375.13 9.93 6.30 25.839 16.393 87380 87380

And now the OFED 1.2 bits I installed overtop of the 1.1 stuff which shipped 
with RHEL5

RHEL5 rx2660 to rx2660, AD313A, OFED 1.2 GA software, TCP_STREAM
1Mx64K. CPU 0 taking interrupts, IB switch in place:

[root at hpcpc106 ~]# for i in 0 1; do for j in 0 1; do echo $i $j `netperf -P 0 -T 
$i,$j -t TCP_STREAM -H 192.168.1.107 -c -C -l 30 -i 30,3 -- -s 1M -S 1M -m 64K`; 
done; done
0 0 2097152 2097152 65536 30.00 5227.08 6.19 25.00 0.388 1.568
0 1 2097152 2097152 65536 30.00 5449.90 6.47 26.77 0.389 1.610
1 0 2097152 2097152 65536 30.00 5235.90 6.70 25.01 0.420 1.565
1 1 2097152 2097152 65536 30.00 5511.77 7.16 26.80 0.426 1.593 *

RHEL5 rx2660 to rx2660, AD313A, OFED 1.2 GA software, bidirectional
TCP_RR 1Mx64Kx12, CPU 0 taking interrupts, IB switch in place:

[root at hpcpc106 ~]# for i in 0 1; do for j in 0 1; do echo $i $j `netperf -P 0 -T 
$i,$j -t TCP_RR -H 192.168.1.107 -c -C -l 30 -i 30,3 -- -s 1M -S 1M -r 64K -b 
12`; done; done
0 0 2097152 2097152 65536 65536 30.00 5314.44 16.13 16.08 121.431 121.049 
2097152 2097152
0 1 !!! WARNING !!! Desired confidence was not achieved within the specified 
iterations. !!! This implies that there was variability in the test environment 
that !!! must be investigated before going further. !!! Confidence intervals: 
Throughput : 0.3% !!! Local CPU util : 20.4% !!! Remote CPU util : 48.2% 2097152 
2097152 65536 65536 30.00 5384.71 17.24 23.42 128.082 174.245 2097152 2097152
1 0 2097152 2097152 65536 65536 30.00 5388.18 17.06 16.27 126.619 120.784 
2097152 2097152
1 1 !!! WARNING !!! Desired confidence was not achieved within the specified 
iterations. !!! This implies that there was variability in the test environment 
that !!! must be investigated before going further. !!! Confidence intervals: 
Throughput : 0.3% !!! Local CPU util : 45.3% !!! Remote CPU util : 0.3% 2097152 
2097152 65536 65536 30.00 5469.22 22.58 17.08 165.328 124.947 2097152 2097152 *

RHEL5 rx2660 to rx2660, AD313A, OFED 1.2 GA software, TCP_RR, CPU 0
taking interrupts, IB switch in place:

[root at hpcpc106 ~]# for i in 0 1; do for j in 0 1; do echo $i $j `netperf -P 0 -T 
$i,$j -t TCP_RR -H 192.168.1.107 -l 30 -i 30,3`; done; done
0 0 87380 87380 1 1 30.00 18990.16 87380 87380 *
0 1 87380 87380 1 1 30.00 14985.03 87380 87380
1 0 87380 87380 1 1 30.00 15045.17 87380 87380
1 1 87380 87380 1 1 30.00 12408.56 87380 87380

(I didn't bother asking for CPU util in the single-byte TCP_RR tests because I 
knew that the confidence intervals wouldn't be met and it would only lengthen 
the runtime)

Sorry that the confidence interval warnings make things hard to read there.

rick jones

> 
> Thanks
> Shirley