[openib-general] ipoib mtu problem with UDP

[Moni Shoua]

Michael S. Tsirkin wrote:

>I tried using ifconfig to limit the ipoib mtu.
>Once I do this on *either* both server and client, or only on the client side,
>UDP seems to stop working:
>
>#ifconfig ib0 mtu 512
>#netperf -c -C -H 11.4.3.68 -f M -t UDP_STREAM
>UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 11.4.3.68
>(11.4.3.68) port 0 AF_INET : demo
>Socket  Message  Elapsed      Messages                   CPU      Service
>Size    Size     Time         Okay Errors   Throughput   Util     Demand
>bytes   bytes    secs            #      #   MBytes/sec % SS     us/KB
>
>118784   65507   10.00       27582      0      172.2     26.33    inf
>118784           10.00           0               0.0     23.40    inf
>
>Things work fine if the mtu on the client side is 2044:
># ifconfig ib0 mtu 2044
># netperf -c -C -H 11.4.3.68 -f M -t UDP_STREAM
>UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 11.4.3.68 (11.4.3.68) port 0 AF_INET : demo
>Socket  Message  Elapsed      Messages                   CPU      Service
>Size    Size     Time         Okay Errors   Throughput   Util     Demand
>bytes   bytes    secs            #      #   MBytes/sec % SS     us/KB
>
>118784   65507   10.00       78488      0      490.1     25.31    2.310
>118784           10.00       68534             428.0     24.55    2.241
>
>Tested with kernel 2.6.19-rc4 and netperf 2.4.2.
>
>  
>
I get the same  results with iperf.
However they succeed with smaller datagrams (netperf uses 65507 by default)

dodly5:/home/shared/testing-tools/x86_64/netperf/netperf-2.4.1 # 
ifconfig ib0
ib0       Link encap:UNSPEC  HWaddr 
00-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
          inet addr:192.168.11.235  Bcast:192.168.11.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:512  Metric:1
          RX packets:42 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14077513 errors:0 dropped:5 overruns:0 carrier:0
          collisions:0 txqueuelen:128
          RX bytes:5776 (5.6 Kb)  TX bytes:6717604780 (6406.4 Mb)

dodly5:/home/shared/testing-tools/x86_64/netperf/netperf-2.4.1 # 
./netperf   -H 192.168.11.233  -t UDP_STREAM -- -m 30000
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 
192.168.11.233 (192.168.11.233) port 0 AF_INET
Socket  Message  Elapsed      Messages
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

262144   30000   10.00       52533      0    1260.59
262144           10.00       22956            550.86


dodly5:/home/shared/testing-tools/x86_64/iperf-2.0.2 # ./iperf -uc 
192.168.11.233 -l 65000
------------------------------------------------------------
Client connecting to 192.168.11.233, UDP port 5001
Sending 65000 byte datagrams
UDP buffer size:   256 KByte (default)
------------------------------------------------------------
[  3] local 192.168.11.235 port 32769 connected with 192.168.11.233 port 
5001
[  3]  0.0-10.9 sec  1.36 MBytes  1.05 Mbits/sec
[  3] Sent 22 datagrams
[  3] WARNING: did not receive ack of last datagram after 10 tries.
dodly5:/home/shared/testing-tools/x86_64/iperf-2.0.2 # ./iperf -uc 
192.168.11.233
------------------------------------------------------------
Client connecting to 192.168.11.233, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size:   256 KByte (default)
------------------------------------------------------------
[  3] local 192.168.11.235 port 32769 connected with 192.168.11.233 port 
5001
[  3]  0.0-10.0 sec  1.25 MBytes  1.05 Mbits/sec
[  3] Sent 893 datagrams
[  3] Server Report:
[  3]  0.0-10.0 sec  1.25 MBytes  1.05 Mbits/sec  0.002 ms    0/  893 (0%)