[ofa-general] Enabling IP_CM warns about multicast packet drops

[Or Gerlitz]

Sumeet Lahorani wrote:
> When we enable IB connected mode and increase MTU to 65520, we see the following
> kernel: ib0: enabling connected mode will cause multicast packet drops
> kernel: ib0: mtu > 2044 will cause multicast packet drops.

> Should we not be doing this? What kind of multicast packets will be dropped?
> If we are not using multicast, do any drivers (bonding, ipoib etc) internally use 
> multicast in a way that will cause them to not work correctly in connected mode? 

Connected mode is supported only for unicast traffic where multicast traffic keeps going over the IB UD QP whose MTU is much lower (e.g 2-4K). To close the gap between the MTU seen by the network stack to the MTU used by the UD QP, IPoIB emulates receiving an icmp packet that tells the os stack to use a different path mtu for this multicast neighbour, see

ipoib_start_xmit --> 
  ipoib_send --> 
   ipoib_cm_skb_too_long(mcast_mtu) --> 
    skb->dst->ops->update_pmtu(skb->dst, mtu)

When IP multicast is not used, multicast is used by the network stack and bonding just for the sake of sending ARPs on the broadcast group, and IGMP where the size of both is way below the IB mtu.

Or.