[ofa-general] Re: [PATCH v3 for 2.6.24] IB/ipoib: enable IGMP for userpsace multicast IB apps

[Or Gerlitz]

Roland Dreier wrote:
> OK, I will merge this for 2.6.24.  However, I still don't really
> understand the changelog entry:

>> The kernel IB stack allows (through the RDMA CM) user space multicast applications
>> to interoperate with IP based apps optionally running at a different IP subnet.
>> 
>> To support this inter-op for the case where the receiving party resides at
>> the IB side, there is a need to handle IGMP (reports/queries) else the local
>> IP router would not forward multicast traffic towards the IB network.

> So in other words you have a userspace app that joins an IPoIB
> multicast group and then it has to do an IP_ADD_MEMBERSHIP socket
> option to trigger IGMP messages being sent out, so that traffic gets
> routed to it?

yes

>> This patch does a lookup on the database used for multicast reference counting and
>> enhances IPoIB to ignore multicast group which is already handled by user space, all
>> this under a per device policy flag. That is when the policy flag allows it, IPoIB
>> will not join and attach its QP to a multicast group which has an entry on the database.

> And then you don't want the kernel IPoIB driver to actually join the
> multicast group for the IP multicast group you added with
> IP_ADD_MEMBERSHIP?  Why is that exactly -- this is the part I'm
> especially hazy on.

Yes I don't want ipoib to receive packets from this group, so it need 
not join/attach to it through the flow of the net core calling ipoib's 
set_multicast_list callback. In the case of IGMP v2 where reports are 
sent over the actual group, IPoIB does join but as "send only", I have 
validated this to work fine with my patch.

The whole idea is that there's a userspace app that joins through the 
rdma-cm and attaches its user space QP to this MGID such that it will 
receive this multicast group packets. Opening a socket and calling add 
membership on it is done since this is the only means to cause the 
kernel to issue IGMP reporting etc on this group.

Other than that IPoIB need not join/attach to this group, doing so on my 
system (*) cuts the performance by half. When I attach two user space 
processes to the same group performance is cut by only ~10%, so the 50% 
drop might turn to be network stack issue or firmware issue or 
combination of both and other things.

At the bottom line, the umcast flag allow users who need to interop with 
IP routers, to signal IPoIB that they don't want groups joined for user 
space receiving to be joined/attached by the kernel.

are we done?

Or.

(*) Mellanox Arbel memfull hca (device 25208), firmware 4.7.600