[ofa-general] Re: [PATCH RFC v2] IB/ipoib: enable IGMP for userpsace multicast IB apps

[Or Gerlitz]

Roland Dreier wrote:
>  > Since both the rdma-cm and ipoib are consumers of the core mutlicast
>  > management code (core/multicast.c which is linked into ib_sa.ko), and the
>  > app (through the rdma-cm) --first--  inserts a record into the database and
>  > only then issues the setsockopt call, if ipoib has a hit on a group it was
>  > told to join, this group must be offloaded by the rdma-cm consumer.
> 
> I'm not sure I understand why that follows.  Couldn't there be some
> other kernel or userspace entity that caused the record to be added?

Actually, no:

The multicast database is managed by <device, port-num, mgid> AND the 
bunch of attributes associated with sa multicast join request. So if two 
consumers attempt to join with the same <device,port,mgid> tuple, and 
the attributes present in the struct ib_sa_mcmember_rec provided to 
ib_sa_join_multicast() are different, the second join fails with -EINVAL 
status, see the comp_rec() function in core/multicast.c.

The rdma-cm first calls ib_sa_get_mcmember_rec() with the partition 
broadcast mgid such it for IPOIB ID's it can use the same attributes 
provided by ipoib for the mgid to join to (for non IPOIB ID's it uses a 
different qkey and mgid, see cma_join_ib_multicast() and cma_set_mgid() 
in core/cma.c).

So indeed the assumption in the patch is that mgids which translate to 
legal IP multicast addresses are inserted into the database either by 
ipoib or rdma-cm consumers who use IPOIB_PS for their ID's.

>  > The per device flag is initialized by the module param value at
>  > ipoib_dev_init()
> 
> I still don't really get why there's a module parameter to set the
> initial value of a flag that only root can change anyway.  Why not
> just the flag through sysfs after loading ipoib rather than having a
> module parameter to do the same thing?

A module param enables adding a

options ib_ipoib umcast_allowed=1

line to /etc/modprobe.conf to make this setting persistent across module 
unload/load (eg reboots) and be applied to all the devices created by 
ipoib. A sysfs entry has to be explicitly written following each device 
creation. The reason for the sysfs entry existence is to allow reading 
the current value and do runtime changes where in environment that the 
admin wants user-mode multicast to be allowed as the default they will 
use the module param, does this make sense?

The reason for the set_ entry to be allowed for writing to root only is 
since we don't want non privilege user to change the settings.

Or.