[openib-general] IPoIB interface for unauthorized partition

[Eitan Zahavi]

Moni Levy wrote:
> On 4/23/06, Eitan Zahavi <eitan at mellanox.co.il> wrote:
> 
>>Hi Moni,
>>
>>Sorry it took me a while to get back to you (was out on vacation ...)
>>
>>Moni Levy wrote:
>>
>>>On 4/10/06, Eitan Zahavi <eitan at mellanox.co.il> wrote:
>>>
>>>
>>>>Hi Hal,
>>>>
>>>>
>>>>
>>>>>-----Original Message-----
>>>>>From: Hal Rosenstock [mailto:halr at voltaire.com]
>>>>>Sent: Monday, April 10, 2006 2:00 PM
>>>>>To: Eitan Zahavi
>>>>>Cc: Roland Dreier; openib-general at openib.org
>>>>>Subject: Re: [openib-general] IPoIB interface for unauthorized
>>>>
>>>>partition
>>>>
>>>>
>>>>>Hi Eitan,
>>>>>
>>>>>On Mon, 2006-04-10 at 02:35, Eitan Zahavi wrote:
>>>>>
>>>>>
>>>>>>Hi Roland,
>>>>>>
>>>>>>Roland Dreier wrote:
>>>>>>
>>>>>>
>>>>>>>   Eitan> I thought the intent of the IB spec when defining P_Key
>>>>>>>   Eitan> index usage (and not P_Key value) was that the P_Key
>>>>
>>>>values
>>>>
>>>>
>>>>>>>   Eitan> would never need to be known above the driver level.
>>>>
>>>>To
>>>>
>>>>
>>>>>>>   Eitan> avoid exposing the P_Key values we could use P_Key
>>>>
>>>>index
>>>>
>>>>
>>>>>>>   Eitan> for creating the IPoIB interfaces.
>>>>>>>
>>>>>>>   Eitan> Does it make sense to work on a patch that would setup
>>>>>>>   Eitan> IPoIB interfaces by the P_Key index (and not by P_Key
>>>>>>>   Eitan> value)?
>>>>>>>
>>>>>>>I don't see how this is feasible.  The index that a particular
>>>>
>>>>P_Key
>>>>
>>>>
>>>>>>>lands at is completely undetermined -- if two nodes wanted to talk
>>>>
>>>>on
>>>>
>>>>
>>>>>>>partition 0x8001 say, how does one know which interface to use
>>>>
>>>>without
>>>>
>>>>
>>>>>>>knowing the index of that P_Key?
>>>>>>
>>>>>>OK, I get it. Actually the way IPoIB defines the broadcast group
>>>>
>>>>MGID exposes
>>>>
>>>>
>>>>>P_Key anyway.
>>>>>
>>>>>
>>>>>>>   Eitan> Also I think the expected behavior for IPoIB should be
>>>>
>>>>that
>>>>
>>>>
>>>>>>>   Eitan> IPoIB "child" interfaces should be "automatically"
>>>>>>>   Eitan> initialized by the code that brings up the interface
>>>>>>>   Eitan> (ifconfig scripts). All valid IPoIB partitions (valid =
>>>>>>>   Eitan> have corresponding broadcast groups) should be
>>>>>>>   Eitan> initialized. By doing so we provide a centralized
>>>>
>>>>control
>>>>
>>>>
>>>>>>>   Eitan> of the partitions and their IPoIB interfaces through
>>>>
>>>>the
>>>>
>>>>
>>>>>>>   Eitan> SM.
>>>>>>>
>>>>>>>Not sure if this is so.  I may want a partition strictly for
>>>>
>>>>storage
>>>>
>>>>
>>>>>>>traffic something like that, so it doesn't make sense to create an
>>>>>>>IPoIB interface for that partition.
>>>>>>
>>>>>>OpenSM provides this capability in the "partition policy":
>>>>>>Each partition is marked explicitly if to be used for IPoIB or not.
>>>>>>So through one file one could actually control the IPoIB interfaces
>>>>>>that will exist in the subnet.
>>>>>
>>>>>The end node does not know the SM policy for that partition though.
>>>>>
>>>>>
>>>>>
>>>>>>My intent is to write some extension to ifup for IPoIB such that all
>>>>
>>>>sub
>>>>
>>>>
>>>>>>interfaces will be automatically started (based on pre-availability
>>>>
>>>>of IPoIB
>>>>
>>>>
>>>>>>broadcast MGID).
>>>
>>>
>>>I'm not sure how ifup is related to that. From what I understand you'd
>>>like ipoib driver to behave as follows:
>>>
>>>1. Get an event ( or figure it out) when a new PKEY is added to the
>>>relevant port partition table.
>>
>>I prefer not to rely on new events. Instead I would like to rely on existing IB Notices:
>>If we register to multicast group create/delete events (traps 66/67) IPoIB can know about each new partition created.
> 
> 
> I'm not sure that this is a good idea, because that way all of the
> IPoIB nodes will get that event and try to join every new MC group and
> partitioning by definition is good for separating a fabric. I think
> that the right thing should be that only the relevant nodes try to
> join the specific MCG.

A node that does not have a P_Key matching the multicast group would not receive the Report anyway.
So there is no problem. If a node is not part of the partition it will never know about the new group.

> 
> 
>>>2. Try to join that new MC group with the MGID it created according to
>>>the PKEY and the spec.  (or maybe query for the MC group existance but
>>>that's not atomic)
>>
>>Simply join the group. We rely on these groups to be pre-created by the SM enforcing policy dictating with partitions should
>>be used for IPoIB and which not.
> 
> 
> If you let all the IPoIB nodes join every new group without checking
> their PKEY tables first, they may even get joined if the SM is not
> eforcing MCG to port policy.
> Is that your plan ?
A compliant SM will never let them join and never report any new group to ports not in the partition.
> 
> 
>>>3. In case it fails nothing is done (no relevant MC group was
>>>pre-created in the SM).
>>
>>Exactly
>>
>>
>>>4. In case it succeeds a new interface is created.
>>>
>>>Is that what you meant ?
>>>
>>>- Moni
>>>
>>>
>>>
>>>>>If that were to be done, it would be cleanest if the child IPoIB
>>>>>interface was created only if that IPoIB broadcast group for that
>>>>>partition exists.
>>>>
>>>>[EZ] This is exactly what I had in mind.
>>>>
>>>>
>>>>>-- Hal
>>>>>
>>>>>
>>>>>
>>>>>>>- R.
>>>>>>>
>>>>>>
>>>>_______________________________________________
>>>>openib-general mailing list
>>>>openib-general at openib.org
>>>>http://openib.org/mailman/listinfo/openib-general
>>>>
>>>>To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>>>>
>>
>>