[openib-general] multicast code/merge status

[Or Gerlitz]

Sean Hefty wrote:
>> sure, it can use the rdmacm qkey (0x1234567 etc) when it creates the 
>> QP and later --if-- the user joins a multicast group modify the qp 
>> state with the group qkey and report it in the cma event such that the 
>> consumer of the rdmacm would set this into his IB UD TX WR

> Changing the qkey would break its existing UD communication.

OK, so we have three use cases here for a UD QP

+1 used only for unicast
+2 used only for multicast
+3 used for both unicast and multicast

and my suggestion (default qkey, when join is completed do qp modify 
with the group qkey) would work fine for use cases

1 - since the user never joins to anything and
2 - same as it works in ipoib

so we are left with use case 3.

To make things simple, the solution i suggest is that that the RDMA CM 
would --not-- do this modify QP/QKEY (that is would set the 0x12345678 
qkey on the modify qp to init) and rather leave it to the RDMA CM 
consumer --if-- they wish to do so. However it will use the ipv4 
broadcast group qkey for doing mcast joins and report this qkey to the 
user in the ud param of the event.

So users that don't care about their qkey would never bother to do this 
modify qp and users who do care would do it and have to take caution if 
their QP is of type 3 (both unicast and mcast).

If you don't like this direction, your idea from below to have two 
option for group type - rdmacm or ipoib and have the consumer specify 
it, so for group type ipoib you will use the ipv4 brd qkey for both join 
and modify qp and for group type rdmacm you would just use the rdmacm 
qkey and do no modify qp - this is fine for us as well.

>> Bottom line, Looking in the IB SPEC and IPoIB RFC i did not see 
>> mentioning of privileged QKEY.
> 
>  From RFC 4391 (ipoib RFC), 4.1:
> 
>  2. Q_Key
> 
>           It is RECOMMENDED that a controlled Q_Key be used with the
>           high-order bit set.  This is to prevent non-privileged
>           software from fabricating and sending out bogus IP datagrams.
> 
> I don't know what qkey is actually assigned, however.

this (what qkey is assigned to the ipv4 broadcast group by different 
SAs) is orthogonal to the discussion we do here.

> I have some path forward related tasks that I would like to complete 
> before starting on this.  I hope to finish that before the end of this 
> week.  I don't want to rush on the multicast support and miss 
> something.  For the rdma cm, we may need to let the user set some 
> options when joining a multicast group.  Maybe something like: join type 
> (send-only or send-receive), group type (ipoib or rdma defined), etc.

As I see it, the group type (or having no types and being always 
interoperable with ipoib as i suggest above) seems easy to add to the 
current implementation and would put it in acceptable state for upstream 
pushing to 2.6.21 and inclusion in OFED 1.2 .

As for the join type, as i told you before, I think it should --not-- 
delay the upstream nor the ofed 1.2 push - if you have the time add this 
  to the user/kernel ABI and have ucma kernel return -EINVAL if someone 
attempts to to send-only join. And if you don't have the time for that, 
it can be added later.

Actually, as you can see in the ipoib code, it never does a 
send-only-non-member join, so my take here is that till the ipoib issue 
is resolved there is no reason to have this complexity in the rdmacm.

> I do plan on requesting that the core multicast changes to ib_sa and 
> ib_ipoib be pulled into 2.6.21.

This is great news but again I think the "nobody perfect" rule applies 
well here, the current rdmacm multicast support (which the little fixes 
we discuss over this thread) can be pushed to 2.6.21 and be enhanced later.

Or.