[openib-general] multicast code/merge status

[Or Gerlitz]

On 1/10/07, Sean Hefty <mshefty at ichips.intel.com> wrote:
>> How about adding sendonly param to the ABI and having the ucma kernel
>> code returning -EINVAL if someone tries to set it to true. Such code can
>> be pushed to 2.6.21 and when you have the time to complete the
>> implementation you can complete this?

> I don't think adding this is a huge deal; I just haven't gotten to it yet.
> However, I'd like to make sure there's enough time once the change is made to
> verify that we have the right result before pushing it upstream.

Compared to the amount of functionality (ib_sa ref mcast counting and
rdma cm u/k mcast API) provided by this code, i really think it can be
done in two phases, first push a code that does not support sendonly
and later once its implemented push a patch that adds this.

Anyway, per the best of my knowledge you would not be able to fully
verify the "right" result , i understand the opensm does not really
support a sendonly join (*) in the sense of that it does not support
configuring "one-way" MFTs for a subset of the branches/leaves of the
multicast group spanning tree, but this is orthogonal to your code.

(*) there are some more issues here which need to be addressed, see
for example the "Some SMs don't support send-only yet" weird comment
at ipoib_mcast_sendonly_join()

> Woody has also seen this issue.  And of course, I can't reproduce it on my
> systems, but I'm actively looking into the problem.  It looks like some sort of
> issue with ipoib trying to join a non-existent multicast group.

mmm, weird, are there active rdmacm multicast consumers involved in
this settings?

>> Looking on the code, i understand that if an multicast consumer attempts
>> to join a group for which another consumer is already joined then it
>> just gets the group params, that is the mgid is your discriminator (with
>> the exception of an all zeros mgid which has a different treatment)
>> which makes much sense to me.

> Not exactly.  The rdma_cm consumer gets the group parameters for the ipoib
> broadcast group.  It uses this information as a template for joining new groups.

OK, got you at last (sorry but i have somehow ignored the call to
ib_addr_get_mgid() at the rdmacm code). So to achieve interop with
IPoIB all we need to do is remove the rdmacm signature bit and not to
over-write the rdmacm qkey on the the qkey of the ipoib ipv4 broadcast
group, are you ok with that?

> One issue is that an rdma_cm consumer can first allocate a UD QP to use with UD
> traffic.  When it later joins a multicast group, the qkey must be the same.  How
> does ipoib handle this?

Well, ipoib first sets a zero qkey into its qp in ipoib_init_qp() and
later in ipoib_mcast_attach() does another qp modify providing
priv->qkey which is the ipv4 broadcast group one, the rdma_cm can
mimic this behaviour for qps created with rdma_create_qp and also for
those created by the user.

>> Since for our apps needs we do intend to join the 224.0.0.1 group,
>> resolving a) above is fine for us --> we will join 224.0.0.1 above,
>> provide the qkey to the rdma cm and it will join to the other group (eg
>> 224.5.5.5) with this qkey.

> I'm not completely following you on this yet.

forget this, there is no need in such tricks since the rdmacm has the
ipv4 broadcast group qkey.

>> can you remind me what the idea/trick here, aren't you supposed to
>> generate an mgid for this case?

> This either returns an existing MCMemberRecord that this node has joined, or it
> fills out an MCMemberRecord that can be used to join a new group.  If the mgid
> is zero, the SA will assign one.

thanks

Or.