[ofa-general] Multicast Performance

[Marcel Heinz]

Hi,

Dotan Barak wrote:
> Marcel Heinz wrote:
>> [low multicast throughput of ~250MB/s with own benchmark tool]
>
> 1) I know that ib_send_bw supports multicast as well, can you please
> check that you can reproduce your problem
> on this benchmark too?

Well, the last time I've checked this, ib_send_bw didn't support
multicast, but this was some months ago. That multicast support seems
a bit odd, since it doesn't create/join the multicast groups and there
is still a 1:1 TCP connection used to establish the IB connection, so
one cannot benchmark "real" multicast scenarios with more than one
receiver.

However, here are the results (I just used ipoib to let it create some
multicast groups for me):

| mh at mhtest0:~$ ib_send_bw -c UD -g mhtest1
| ------------------------------------------------------------------
|                     Send BW Multicast Test
| Connection type : UD
| Max msg size in UD is 2048 changing to 2048
| Inline data is used up to 400 bytes message
|   local address:  LID 0x01, QPN 0x4a0405, PSN 0x8667a7
|   remote address: LID 0x03, QPN 0x4a0405, PSN 0x5d41b6
| Mtu : 2048
| ------------------------------------------------------------------
|  #bytes #iterations    BW peak[MB/sec]    BW average[MB/sec]
|    2048        1000             301.12                247.05
| ------------------------------------------------------------------

This is the same result as my own benchmark showed in that scenario.

> 2) You should expect that multicast messages will be slower than
> unicast because the HCA/switch treat them in different way
> (message duplication need to be done if needed).

Yes, but 250MB/s vs. 1100MB/s (UD unicast throughput) seems to be a bit
too much of overhead, don't you think? Especially if I take into account
that with my own benchmark, I can get ~950MB/s when I start another
receiver on the same host as the sender. Note that both of the
receivers, the local and the remote one, are seeing all packets at that
rate, so the HCAs and the switch must be able to handle multicast
packets with this throughput.

The other strange thing is that multicast traffics slows down other
traffic way more than the bandwith it consumes. Moreover, it seems like
it limits any other connections to the same throughput than that of the
multicast traffic, which looks suspicious to me.

The same behavior can be reproduced with ib_send_bw, by starting an
unicast and multicast run in parallel:

| mh at mhtest0:~$ ib_send_bw -c UD mhtest1 & ib_send_bw -c UD -g mhtest1\
| -p 18516
| ./ib_send_bw -c UD -g mhtest1 -p 18516
| [1] 4927
| ------------------------------------------------------------------
|                     Send BW Test
| Connection type : UD
| Max msg size in UD is 2048 changing to 2048
| ------------------------------------------------------------------
|                     Send BW Multicast Test
| Connection type : UD
| Max msg size in UD is 2048 changing to 2048
| Inline data is used up to 400 bytes message
| Inline data is used up to 400 bytes message
|   local address:  LID 0x01, QPN 0x530405, PSN 0xe98523
|   local address:  LID 0x01, QPN 0x530406, PSN 0x3b338e
|   remote address: LID 0x03, QPN 0x540405, PSN 0x5c53e2
| Mtu : 2048
|   remote address: LID 0x03, QPN 0x540406, PSN 0xff883f
| Mtu : 2048
| ------------------------------------------------------------------
|  #bytes #iterations    BW peak[MB/sec]    BW average[MB/sec]
| ------------------------------------------------------------------
|  #bytes #iterations    BW peak[MB/sec]    BW average[MB/sec]
|    2048        1000             692.41                270.26
|    2048        1000             246.00                244.68
| ------------------------------------------------------------------
| ------------------------------------------------------------------

Doing 2 unicast UD runs in parallel, I'm getting ~650MB/s average
bandwith for each, which sounds reasonable.

Also, when using  bidircetional mode, I'm getting ~1900MB/s (amlost
doubled) throughput for unicast, but still ~250MBs for multicast.

Regards,
Marcel