[openib-general] multicast code/merge status
ogerlitz at voltaire.com
Wed Jan 17 03:00:41 PST 2007
Sean Hefty wrote:
>> sure, it can use the rdmacm qkey (0x1234567 etc) when it creates the
>> QP and later --if-- the user joins a multicast group modify the qp
>> state with the group qkey and report it in the cma event such that the
>> consumer of the rdmacm would set this into his IB UD TX WR
> Changing the qkey would break its existing UD communication.
OK, so we have three use cases here for a UD QP
+1 used only for unicast
+2 used only for multicast
+3 used for both unicast and multicast
and my suggestion (default qkey, when join is completed do qp modify
with the group qkey) would work fine for use cases
1 - since the user never joins to anything and
2 - same as it works in ipoib
so we are left with use case 3.
To make things simple, the solution i suggest is that that the RDMA CM
would --not-- do this modify QP/QKEY (that is would set the 0x12345678
qkey on the modify qp to init) and rather leave it to the RDMA CM
consumer --if-- they wish to do so. However it will use the ipv4
broadcast group qkey for doing mcast joins and report this qkey to the
user in the ud param of the event.
So users that don't care about their qkey would never bother to do this
modify qp and users who do care would do it and have to take caution if
their QP is of type 3 (both unicast and mcast).
If you don't like this direction, your idea from below to have two
option for group type - rdmacm or ipoib and have the consumer specify
it, so for group type ipoib you will use the ipv4 brd qkey for both join
and modify qp and for group type rdmacm you would just use the rdmacm
qkey and do no modify qp - this is fine for us as well.
>> Bottom line, Looking in the IB SPEC and IPoIB RFC i did not see
>> mentioning of privileged QKEY.
> From RFC 4391 (ipoib RFC), 4.1:
> 2. Q_Key
> It is RECOMMENDED that a controlled Q_Key be used with the
> high-order bit set. This is to prevent non-privileged
> software from fabricating and sending out bogus IP datagrams.
> I don't know what qkey is actually assigned, however.
this (what qkey is assigned to the ipv4 broadcast group by different
SAs) is orthogonal to the discussion we do here.
> I have some path forward related tasks that I would like to complete
> before starting on this. I hope to finish that before the end of this
> week. I don't want to rush on the multicast support and miss
> something. For the rdma cm, we may need to let the user set some
> options when joining a multicast group. Maybe something like: join type
> (send-only or send-receive), group type (ipoib or rdma defined), etc.
As I see it, the group type (or having no types and being always
interoperable with ipoib as i suggest above) seems easy to add to the
current implementation and would put it in acceptable state for upstream
pushing to 2.6.21 and inclusion in OFED 1.2 .
As for the join type, as i told you before, I think it should --not--
delay the upstream nor the ofed 1.2 push - if you have the time add this
to the user/kernel ABI and have ucma kernel return -EINVAL if someone
attempts to to send-only join. And if you don't have the time for that,
it can be added later.
Actually, as you can see in the ipoib code, it never does a
send-only-non-member join, so my take here is that till the ipoib issue
is resolved there is no reason to have this complexity in the rdmacm.
> I do plan on requesting that the core multicast changes to ib_sa and
> ib_ipoib be pulled into 2.6.21.
This is great news but again I think the "nobody perfect" rule applies
well here, the current rdmacm multicast support (which the little fixes
we discuss over this thread) can be pushed to 2.6.21 and be enhanced later.
More information about the general