[ofw] IPoIB-CM MTU and bandwidth scaling issue - TCP only, UDP OK.

[Smith, Stan]

Hello,
  I've been observing strange TCP bandwidth performance behaviors with varying IPoIB-CM MTU sizes.
For a fixed ttcp/TCP transfer size (-n 2048 -l 8192), one would expect bandwidth to increase as MTU size increases.

Using a heavily debug/checked IPoIB-CM driver on twin Windows 2008 R2 x64 systems with Infinihost HCAs (MT25208), the observed behavior is:

When the MTU is > 8KB ... 64KB, TCP performance (ttcp -n 2048 -l 8192) is around 0.04 MB/sec?

When the MTU is 2048 ... 8KB, TCP performance scales from 85 to 113 MB/sec.

 MTU (bytes)             sender's BW MB/sec
 -------------------------------------
 2048   2K                        85.11
 4096   4K                       102.56
 8192   8K                       113.48
 9216   9K                       000.04
 12288 12K                       000.04
 16384 16K                       000.04
 65536 64K                       000.04

 65536 64K (UDP)                 128.00

My local definition of Round-Trip Time (RTT) is defined as the time measured from a TCP segment/packet send (ib_post_send) to the time when the TCP ACK arrives as viewed from Wireshark.

When the IPOIB-CM MTU <= 8KB, such that the NDIS send SGL (scatter gather list) size is <= 8KB, RTT is around 90 us (microseconds).

When the IPoIB-CM MTU is > 8KB, RTT climbs to around 203 ms (milliseconds); ouch!

The same ttcp transfer size using UDP instead of TCP does not suffer the performance drop @ 8KB.

Are you aware of any TCP protocol settings/properties which could/should be adjusted to bring TCP bandwidth performance up to or > 8KB MTU BW levels?

thanks,

Stan.