[openib-general] IPoIB interface for unauthorized partition

[Moni Levy]

On 4/23/06, Eitan Zahavi <eitan at mellanox.co.il> wrote:
> Hi Moni,
>
> Sorry it took me a while to get back to you (was out on vacation ...)
>
> Moni Levy wrote:
> > On 4/10/06, Eitan Zahavi <eitan at mellanox.co.il> wrote:
> >
> >>Hi Hal,
> >>
> >>
> >>>-----Original Message-----
> >>>From: Hal Rosenstock [mailto:halr at voltaire.com]
> >>>Sent: Monday, April 10, 2006 2:00 PM
> >>>To: Eitan Zahavi
> >>>Cc: Roland Dreier; openib-general at openib.org
> >>>Subject: Re: [openib-general] IPoIB interface for unauthorized
> >>
> >>partition
> >>
> >>>Hi Eitan,
> >>>
> >>>On Mon, 2006-04-10 at 02:35, Eitan Zahavi wrote:
> >>>
> >>>>Hi Roland,
> >>>>
> >>>>Roland Dreier wrote:
> >>>>
> >>>>>    Eitan> I thought the intent of the IB spec when defining P_Key
> >>>>>    Eitan> index usage (and not P_Key value) was that the P_Key
> >>
> >>values
> >>
> >>>>>    Eitan> would never need to be known above the driver level.
> >>
> >>To
> >>
> >>>>>    Eitan> avoid exposing the P_Key values we could use P_Key
> >>
> >>index
> >>
> >>>>>    Eitan> for creating the IPoIB interfaces.
> >>>>>
> >>>>>    Eitan> Does it make sense to work on a patch that would setup
> >>>>>    Eitan> IPoIB interfaces by the P_Key index (and not by P_Key
> >>>>>    Eitan> value)?
> >>>>>
> >>>>>I don't see how this is feasible.  The index that a particular
> >>
> >>P_Key
> >>
> >>>>>lands at is completely undetermined -- if two nodes wanted to talk
> >>
> >>on
> >>
> >>>>>partition 0x8001 say, how does one know which interface to use
> >>
> >>without
> >>
> >>>>>knowing the index of that P_Key?
> >>>>
> >>>>OK, I get it. Actually the way IPoIB defines the broadcast group
> >>
> >>MGID exposes
> >>
> >>>P_Key anyway.
> >>>
> >>>>>    Eitan> Also I think the expected behavior for IPoIB should be
> >>
> >>that
> >>
> >>>>>    Eitan> IPoIB "child" interfaces should be "automatically"
> >>>>>    Eitan> initialized by the code that brings up the interface
> >>>>>    Eitan> (ifconfig scripts). All valid IPoIB partitions (valid =
> >>>>>    Eitan> have corresponding broadcast groups) should be
> >>>>>    Eitan> initialized. By doing so we provide a centralized
> >>
> >>control
> >>
> >>>>>    Eitan> of the partitions and their IPoIB interfaces through
> >>
> >>the
> >>
> >>>>>    Eitan> SM.
> >>>>>
> >>>>>Not sure if this is so.  I may want a partition strictly for
> >>
> >>storage
> >>
> >>>>>traffic something like that, so it doesn't make sense to create an
> >>>>>IPoIB interface for that partition.
> >>>>
> >>>>OpenSM provides this capability in the "partition policy":
> >>>>Each partition is marked explicitly if to be used for IPoIB or not.
> >>>>So through one file one could actually control the IPoIB interfaces
> >>>>that will exist in the subnet.
> >>>
> >>>The end node does not know the SM policy for that partition though.
> >>>
> >>>
> >>>>My intent is to write some extension to ifup for IPoIB such that all
> >>
> >>sub
> >>
> >>>>interfaces will be automatically started (based on pre-availability
> >>
> >>of IPoIB
> >>
> >>>>broadcast MGID).
> >
> >
> > I'm not sure how ifup is related to that. From what I understand you'd
> > like ipoib driver to behave as follows:
> >
> > 1. Get an event ( or figure it out) when a new PKEY is added to the
> > relevant port partition table.
> I prefer not to rely on new events. Instead I would like to rely on existing IB Notices:
> If we register to multicast group create/delete events (traps 66/67) IPoIB can know about each new partition created.

I'm not sure that this is a good idea, because that way all of the
IPoIB nodes will get that event and try to join every new MC group and
partitioning by definition is good for separating a fabric. I think
that the right thing should be that only the relevant nodes try to
join the specific MCG.

>
> > 2. Try to join that new MC group with the MGID it created according to
> > the PKEY and the spec.  (or maybe query for the MC group existance but
> > that's not atomic)
> Simply join the group. We rely on these groups to be pre-created by the SM enforcing policy dictating with partitions should
> be used for IPoIB and which not.

If you let all the IPoIB nodes join every new group without checking
their PKEY tables first, they may even get joined if the SM is not
eforcing MCG to port policy.
Is that your plan ?

>
> > 3. In case it fails nothing is done (no relevant MC group was
> > pre-created in the SM).
> Exactly
>
> > 4. In case it succeeds a new interface is created.
> >
> > Is that what you meant ?
> >
> > - Moni
> >
> >
> >>>If that were to be done, it would be cleanest if the child IPoIB
> >>>interface was created only if that IPoIB broadcast group for that
> >>>partition exists.
> >>
> >>[EZ] This is exactly what I had in mind.
> >>
> >>>-- Hal
> >>>
> >>>
> >>>>> - R.
> >>>>>
> >>>>
> >>_______________________________________________
> >>openib-general mailing list
> >>openib-general at openib.org
> >>http://openib.org/mailman/listinfo/openib-general
> >>
> >>To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> >>
>
>