[ofa-general] Re: Re: [PATCH] for-2.6.23 ib/umad: add partition support

[Hal Rosenstock]

On Thu, 2007-06-21 at 09:51, Michael S. Tsirkin wrote:
> > Quoting Michael S. Tsirkin <mst at dev.mellanox.co.il>:
> > Subject: Re: Re: [PATCH] for-2.6.23 ib/umad: add partition support
> > 
> > > Quoting Sean Hefty <sean.hefty at intel.com>:
> > > Subject: RE: Re: [PATCH] for-2.6.23 ib/umad: add partition support
> > > 
> > > >This assumes an open file desriptor per-pkey, so the proposed API
> > > >extension umad_set_pkey would have to be changed to be per-port rather
> > > >than per-mad. But I think this is a better API, too: most apps
> > > >likely work within a single partition.
> > > 
> > > I don't think this is true for apps that use the userspace MAD interface (e.g.
> > > opensm).
> > 
> > SM (rather, SA) can just open file descriptor per pkey - it created them itself,
> > and there's a small number of partitions.
> > 
> > > Beyond that, this approach doesn't work for receiving MADs on different PKeys.
> > 
> > Yes, it does: we just filter out the MADs where pkey does not match.
> > 
> > I think that most other apps (besides SA) should really treat
> > each partition as a separate network. So getting MADs for a specific
> > pkey, rather than all pkeys, makes total sense to me.
> 
> Hal, could you pls comment on whether this approach will work for opensm?

I will answer at the "high" level rather than some of the details
discussed in previous postings which we may get back to later.

As far as SA is concerned, as all nodes are required to at least support
the limited default partition, the SA uses the full default partition
for communication.

As to other current (and potential) management applications:

PerfMgr will want PMA access on all ports on all nodes. It may also be
constrained to a similar environment as SA (running on a node which
supports the full default partition). If it is not constrained in such a
manner, it needs to be on all partitions in the subnet or it will only
be able to access a portion of the ports in the subnet. That actually
might be a model some might ultimately want.

Diagnostics may be happy with a single partition (or likely the set of
partitions the end node they are running from reside on).

Bottom line is that it can likely work with either model but there are
tradeoffs underneath this "high" level which may not have been
sufficiently explored/discussed as yet.

I'm not sure I like having a different fd per pkey: It's a different
model than currently being used and that would cause more changes to
consumers (as opposed to the other approach) which aren't a clear win to
me (and uses more fds).

-- Hal