[ofa-general] Multi-threaded diags (Was: Re: [PATCH 4/5] infiniband-diags/libibnetdisc: Introduce a context object.)

[Jason Gunthorpe]

On Fri, Sep 18, 2009 at 03:22:22PM -0700, Ira Weiny wrote:

> > main()
> > {
> >   foo = libibnetdisc_setup();
> >   libibnetdisc_discover_all(foo,res);
> >   // Do interesting things with res.
> > } 
> 
> That is the current use case.  However I can see use cases were discover is
> called periodically to get a new snapshot of the fabric.  Also since the
> discover can scan parts of the fabric ("libibnetdisc_discover_part") and
> return a fabric which represents pieces of the whole I could see "fabric"
> operations such as merge, update, and replace.

Sure, the way I've approached this in the past is that the fabric
description is stored as a directect graph, and the usual set of graph
manipulation primitives (BFS, difference, join, splice, etc) are
available to work on it. This makes alot of the stuff people want to
do expressable via quite natural graph concepts.

> > Sure, the entire library is not thread safe around the ibmad_port
> > context. But who cares? If the caller to libibnetdisc wants to thread
> > that way they need to open another context.
> 
> Yes, they can but how do they know they need to do this?
> Furthermore how many context's are required?

Well, that is a doc question right? In C - no metion of threading in
docs == not thread safe.

>  The bottom line is I wanted multiple outstanding queries.  I am not
> going to open a context for each query.  The amount of code required
> to process and sort Transaction ID's should be provided by libibmad
> or a layer at that level.  It should not be required for every user
> process or user lib.  Furthermore my prototype code does not support
> redirect.  Therefore it makes the code even more difficult.  Why
> make every user suffer this problem?

The transaction ID to FD sorting code is provided in the kernel. If
someone wants threads they really want TID to thread mapping so that a
synchronous control flow is prossible:

  madSet(foo,value);  // Sends a MAD, then blocks on a recieve for a TID match'd reply

This is why it is unsuitable for libumad to do any kind of threading
how does it handle multiplexing access to the FD from multiple threads
without a huge, huge mess internally?

> I am a bit confused.  Do you mean to open multiple umad fds such that the
> kernel will do the TID based dispatch for you?  Or are you suggesting a
> different kernel umad implementation?

Yes, that is what I am suggesting.

Every thread you create gets a private FD and a private mad
context. The mad layer is not threaded (beyond being re-entrant). Each
thread sends and blocks on the thread specific FD and the kernel
multiplexes transmits and sorts replies to direct them to the proper
waiting thread.

Anything else is a big can of worms.

But, single threaded event FSM based is almost always dramatically
simpler and faster. But even so, it would work the same way with
threads, each thread gets an thread specific FSM context.

> > Well, the very best way to do this is to have a FSM engine API at the
> > core of the MAD libary:
> >   mad_ctx->callback = done_this;
> >   mad_post(mad,mad_ctx)
> > 
> > done_this(reply):
> >   ...
> 
> Which way do you propose to do this, have a thread calling "done_this" or
> having the user call an event loop?

Look at something like glib to see how this generally works.. but yes,
this is done with a top level poll event loop waiting on the umad
fd. A reasonable goal would to have an FSM interface API that could
be plugged into glib easially.

This gives you mad level parallelism without threads.

> I see some things of value in libibmad.  However, I have been reluctant to use
> it in the past and I agree it needs fixing.  I don't want to reinvent the
> wheel but perhaps that is what needs to be done...

I'm just about to the point where I need something alot better for the
little app I'm working on - I want to setup multipath IB connections,
which means sophisticated PR queries..

So, I'd like to see this fixed up too, and I can probably work on a
few things. We can donate our structure parsing codegen framework
which is dramatically better than what libibmad uses today.

I'm specifically interested in GMP's for PR queries, but in much the
same infrastructure covers both SMPs and GMPs.

What I'd like is a nice uniform language across libibcm and this the
MAD library so I can just do a PR, get the result, pass it to the CM
and up into the kernel without huge app specific code to do all
that. Nothing like that exists right now and it sucks. Very hard to
write IB apps.

Jason