[ofa-general] Multicast Performance

[Hal Rosenstock]

On Thu, 2008-05-29 at 11:19 +0200, Marcel Heinz wrote:
> Hi,
> 
> Dotan Barak wrote:
> > Marcel Heinz wrote:
> >> [low multicast throughput of ~250MB/s with own benchmark tool]
> >
> > 1) I know that ib_send_bw supports multicast as well, can you please
> > check that you can reproduce your problem
> > on this benchmark too?
> 
> Well, the last time I've checked this, ib_send_bw didn't support
> multicast, but this was some months ago. That multicast support seems
> a bit odd, since it doesn't create/join the multicast groups and there
> is still a 1:1 TCP connection used to establish the IB connection, so
> one cannot benchmark "real" multicast scenarios with more than one
> receiver.
> 
> However, here are the results (I just used ipoib to let it create some
> multicast groups for me):
> 
> | mh at mhtest0:~$ ib_send_bw -c UD -g mhtest1
> | ------------------------------------------------------------------
> |                     Send BW Multicast Test
> | Connection type : UD
> | Max msg size in UD is 2048 changing to 2048
> | Inline data is used up to 400 bytes message
> |   local address:  LID 0x01, QPN 0x4a0405, PSN 0x8667a7
> |   remote address: LID 0x03, QPN 0x4a0405, PSN 0x5d41b6
> | Mtu : 2048
> | ------------------------------------------------------------------
> |  #bytes #iterations    BW peak[MB/sec]    BW average[MB/sec]
> |    2048        1000             301.12                247.05
> | ------------------------------------------------------------------
> 
> This is the same result as my own benchmark showed in that scenario.
> 
> > 2) You should expect that multicast messages will be slower than
> > unicast because the HCA/switch treat them in different way
> > (message duplication need to be done if needed).
> 
> Yes, but 250MB/s vs. 1100MB/s (UD unicast throughput) seems to be a bit
> too much of overhead, don't you think?

Agreed.

> Especially if I take into account
> that with my own benchmark, I can get ~950MB/s when I start another
> receiver on the same host as the sender. Note that both of the
> receivers, the local and the remote one, are seeing all packets at that
> rate, so the HCAs and the switch must be able to handle multicast
> packets with this throughput.

Perhaps this is a static rate issue.

What SM is being used ?

-- Hal

> The other strange thing is that multicast traffics slows down other
> traffic way more than the bandwith it consumes. Moreover, it seems like
> it limits any other connections to the same throughput than that of the
> multicast traffic, which looks suspicious to me.
> 
> The same behavior can be reproduced with ib_send_bw, by starting an
> unicast and multicast run in parallel:
> 
> | mh at mhtest0:~$ ib_send_bw -c UD mhtest1 & ib_send_bw -c UD -g mhtest1\
> | -p 18516
> | ./ib_send_bw -c UD -g mhtest1 -p 18516
> | [1] 4927
> | ------------------------------------------------------------------
> |                     Send BW Test
> | Connection type : UD
> | Max msg size in UD is 2048 changing to 2048
> | ------------------------------------------------------------------
> |                     Send BW Multicast Test
> | Connection type : UD
> | Max msg size in UD is 2048 changing to 2048
> | Inline data is used up to 400 bytes message
> | Inline data is used up to 400 bytes message
> |   local address:  LID 0x01, QPN 0x530405, PSN 0xe98523
> |   local address:  LID 0x01, QPN 0x530406, PSN 0x3b338e
> |   remote address: LID 0x03, QPN 0x540405, PSN 0x5c53e2
> | Mtu : 2048
> |   remote address: LID 0x03, QPN 0x540406, PSN 0xff883f
> | Mtu : 2048
> | ------------------------------------------------------------------
> |  #bytes #iterations    BW peak[MB/sec]    BW average[MB/sec]
> | ------------------------------------------------------------------
> |  #bytes #iterations    BW peak[MB/sec]    BW average[MB/sec]
> |    2048        1000             692.41                270.26
> |    2048        1000             246.00                244.68
> | ------------------------------------------------------------------
> | ------------------------------------------------------------------
> 
> Doing 2 unicast UD runs in parallel, I'm getting ~650MB/s average
> bandwith for each, which sounds reasonable.
> 
> Also, when using  bidircetional mode, I'm getting ~1900MB/s (amlost
> doubled) throughput for unicast, but still ~250MBs for multicast.
> 
> Regards,
> Marcel
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general