[openib-general] ipoib & the partial pkey, was: librdmacm: fix bug causing failure to work with partial membership pkey

[Hal Rosenstock]

On Thu, 2007-02-22 at 18:35, Sean Hefty wrote:
> >Doesn't this allow ipoib to join a multicast group for which it may not be able
> >to communicate with all members?  For the broadcast group, this seems like an
> >error to me.  Can ipoib work in such a configuration?  If all nodes were
> >assigned a partial membership PKey, none of them could communicate, but no
> >errors would be generated anywhere.
> 
> I looked into this more...
> 
> RFC 4391 states (middle of page 5):
> 
> For a node to join a partition, one of its ports must be assigned the relevant
> P_Key by the SM [RFC4392].
> 
> Jumping to RFC 4392 (top of page 4):
> 
> at the time of creating an IB multicast group, multiple values such as the
> P_Key, Q_Key, Service Level, Hop Limit, Flow ID, TClass, MTU, etc.  have to be
> specified.  These values should be such that all potential members of the IB
> multicast group are able to communicate with one another when using them.

Seems to me that for P_Key this would mean full membership.

> and page 14:
> 
> Note that this IB_join to the broadcast group is a FullMember join.

FullMember here is referring to MCMemberRecord:JoinState rather than
partition membership.

-- Hal

> If any of
> the ports or the switches linking the port to the rest of the IPoIB subnet
> cannot support the parameters (e.g., path MTU or P_Key) associated with the
> broadcast group, then the IB_join request will fail and the requesting port will
> not become part of the IPoIB subnet
> 
> My initial interpretation of these statements lead me to believe that pkey check
> in ib_find_cached_pkey should not mask out the upper bit, which would prevent
> ipoib from joining a multicast group until it has been configured with the full
> membership pkey for the broadcast group.  Does this seem reasonable?
> 
> - Sean