[openib-general] RE: userspace event reporting

[Fab Tillier]

> From: Libor Michalek [mailto:limichal at cisco.com]
> Sent: Wednesday, July 20, 2005 5:55 PM
> 
> On Wed, Jul 20, 2005 at 05:17:09PM -0700, Fab Tillier wrote:
> > > From: Libor Michalek [mailto:libor at topspin.com]
> > > Sent: Wednesday, July 20, 2005 5:02 PM
> > >
> > >   My question is do we really want this, since an application will likely
> > > have yet another table containing the app specific connection structure.
> > > How that table is locked and managed will differ based on the threading
> > > model. I think it's an incorrect assumption that if we create this level
> > > thread safety in the uCM then the app connection structure can be placed
> > > in a uCM user supplied pointer (e.g. the kCM context variable) and avoid
> > > this type of app level connection table.
> >
> > Ok, now that I think more about it, I think that having the ability to do
> > an async, per-CM ID get_event call would help tremendously.  The
> > application knows what CM IDs it has, and thus can initiate such a
> > get_event request for each. The app can then maintain its own reference
> > count on its internal structures (which is really what is needs to do),
> > rather than rely on the CM providing the reference counting and
> > synchronization.
> 
>   Well, I think having a per-CM ID get_event is a useful thread based
> programming model, but I don't think it's the one and only approach
> for all applications. Again, it's a layer that could (should?) be
> implemented entirely in userspace on top of the current interface.

I'm trying to think how you could implement the per-CM ID get event without
introducing an extra thread that would sit there polling for events, and then
perform the multiplexing.  Maybe having extra threads isn't a big deal, but
that's where the async model comes in handy - a single threaded app can get
exactly the behavior it wants using AIO.

>   Also, I agree with your last sentence. I don't think it necessarily
> follows or depends on the previous two, it's true for all most all
> serious apps, not just if there is a per-CM ID get_event.

Right - it's a matter of the CM enabling the app to do what it needs to do.

> > Is it even possible to make async get_event requests, from a coding
> > perspective? Would the resulting usage model work for clients?  If
> > using an AIO read request, could the file offset could be used to
> > convey the CM ID being polled?
> 
>   Well, you can poll on the file descriptor, which means you can submit
> an AIO poll request. You could submit an AIO read, and you would get
> back the ABI data in your buffer, which contains the cm_id. However, I'm
> not sure what you're getting at.

I don't know how AIO works in Linux.  The idea would be to allow a single
threaded client to issue a get_event AIO request for each CM ID that client has,
and then use the same thread to reap the AIO completions in whatever order they
come in.  This model allows the client to easily keep references on its internal
structures associated with each individual CM ID.

Does that make more sense?  Does Linux have a concept similar to the I/O
completion ports in Windows for gathering AIO requests?

- Fab