[openib-general] First Multicast Leave disconnects all other clients

[Hal Rosenstock]

Hi Eitan,

On Wed, 2005-11-30 at 10:03, Eitan Zahavi wrote:
> Sorry for the late response.
> 
> The bottom line:
> We are missing 3 agents in the OpenIB stack:
> InformInfo - handling registrations and Report dispatching

These are not currently used.

> ServiceRecord - tracks registrations

ServiceRecord is implemented in sa_query (and was used by AT/uAT but
that is largely historical now)

> Multicast Join/Leave - tracking registrations to multicast groups and
> ref-counting 
> 
> All these agents should be able to cleanup dead client registrations and
> also provide re-registration in case of SM ClientReregistration event.

In OpenIB, any Set of PortInfo (which includes ClientReregister)
currently causes a (coarse) event (LID change) which causes IPoIB client
to reregister its multicasts registrations with the SA.

> Please see below
> > >
> > > It seems the IBTA intent was that the IB driver will be responsible
> for maintaining
> > the list of clients
> > > registered to each group.
> > 
> > Yes, the end node is responsible for tracking the registrations within
> > the node and fabricating responses when the node does not want to
> leave.
> > Is delete a different case though ?
> [EZ] No it is not. Delete of multicast group is really the last leave.

There is an explicit delete. While it shouldn't be needed to be forced,
there is always some scenario where this is useful.

> > > But the IB core does not track what clients registered (through SA
> requests) to a
> > particular multicast group.
> > > The first client to leave the group causes the rest (of the clients)
> to be disconnected.
> > 
> > This is an implementation issue IMO and applies to other subscriptions
> > too (not just limited to multicast).
> [EZ] I agree it is an implementation issue. I hope it will get
> implemented in OpenIB.

It will. It's a question of priorities and timing.

> > > My proposal is to provide an API for such registrations at both user
> and kernel and
> > track the requesting processes.
> > > Cleanup is also required both by process and kernel module
> granularity.
> > 
> > Is the API the SA client request itself for this ? Shouldn't the
> > tracking be done there (within sa_query.c) ?
> [EZ] It will be hard to sniff the MADs (especially user level) for all
> the registration flows.

It's not the sniffing which is hard but perhaps identifying which client
(and reference counting).

> So I propose we should have
> ib_join/ib_leave/ib_reg_svc/ib_unreg_svc/ib_reg_inform/ib_unreg_inform.
> Both in user land and in kernel.

I think this is TBD and the API would be discussed on this list first
prior to any implementation.

> > > BTW: The same API could also handle "Client Reregistration" for
> multicast groups,
> > 
> > Client reregistration is for all subscriptions (including
> ServiceRecords
> > and events as well).
> [EZ] Yes exactly. I believe similar problem exists for all
> registrations.
> > 
> > > such that we could avoid the need to have that code duplicated by
> every client.
> > 
> > I'm missing how client reregistration would help here. Can you
> elaborate
> > ?
> [EZ] It is related to the reference tracking: 
> If a kernel module tracks all registrations to refcount them and perform
> cleanup, it could with similar effort also send the - re-registration in
> the event of SM change ...

Sure, there are multiple ways to skin the same cat.

> > 
> > > But this refers to yet another API that is missing: Report
> dispatching which deserves
> > its own
> > > mail...
> > 
> > I'm missing the connection between reregistration and report
> > dispatching.
> [EZ] Sorry for not being verbose. The need for Events dispatcher is
> based on the fact that only one client should respond to Report with
> ReportRepress. Reports are "unsolicited" MADs coming into the device. In
> umad the implementation prevents any "multiple" client registration for
> receiving any "unsolicited" MAD - only one class-agent needs to be there
> handling "unsolicited" messages. This is fine - but what it means is
> that when two clients wants to be notified about events they should
> register with that agent and the agent should be able to dispatch the
> message to all registered clients as well as send only one response
> back.

Wouldn't report represses be reference counted and only actually sent on
the wire when all subscribed clients within the node indicated repress ?

-- Hal