[ofa-general] Direct / "Crossover" IPOIB

[Or Gerlitz]

Robert Dunkley wrote:
> What kind of bandwidth do other users see with TCP on qperf?
using qperf I get the following

rc_bw        1.95 GB/s   
rc_bi_bw    3.9 GB/s
tcp_bw        995 MB/s  - datagram mode mtu 2044
tcp_bw        1.15  GB/s  - connected mode mtu 65520

my systems have ConnectX DDR HCA (25418) and  PCIe gen2

 From your rc_bw results it seems that your systems have gen1 PCIe, and 
from the ibstat output
I see that you have the one port Sinai HCA (25204). The datagram mode 
results with connectx
should be better since it supports TCP stateless offloads such as 
checksum and LSO.

To change the mode of  ipoib use

echo $mode > /sys/class/net/$dev/mode  where $mode is connected or datagram

As for latency, from my experience with qperf it doesn't give the 
optimized latency results, so for verbs based results
I suggest to use ib_write_lat which is provided with the perftest 
package, for tcp latency results, you might want to use netperf with 
TCP_RR and UDP_RR tests, eg:

> netperf -H 10.10.0.90 -D 1, -l 10 -fM -t TCP_RR -- -D L,R -m 4 -M 4
> netperf -H 10.10.0.90 -D 1, -l 10 -fM -t UDP_RR -- -m 4 -M 4
running these on my systems I see 25k (and above) transactions per 
second which suggests an RTT/2 latency of about 20us (40us / 2) you saw 
with qperf...

Or