[ofa-general] Re: Re: [PATCH RFC] sharing userspace IB objects

[Michael S. Tsirkin]

> Quoting Gleb Natapov <glebn at voltaire.com>:
> Subject: Re: Re: [PATCH RFC] sharing userspace IB objects
> 
> On Tue, Jun 26, 2007 at 12:51:25PM +0300, Michael S. Tsirkin wrote:
> > > Quoting Gleb Natapov <glebn at voltaire.com>:
> > > Subject: Re: Re: [PATCH RFC] sharing userspace IB objects
> > > 
> > > On Tue, Jun 26, 2007 at 10:06:41AM +0300, Michael S. Tsirkin wrote:
> > > > >    Is there really a strong use case for making every type of object
> > > > >    shareable?  Can we handle the SRC stuff without going to this
> > > > >    extreme of complexity?
> > > > 
> > > > This is not directly related to SRC: this is an effort
> > > > to make it possible to share QPs, CQ etc across processes
> > > > in the same way as they can be currently shared across threads.
> > > > So assuming that we want multiple processes to post to
> > > > the same QP, how can we support this?
> > > 
> > > Are you absolutely sure you even want to support this?
> > 
> > Take a look here :)
> > http://www.quotedb.com/quotes/1007
> So there is still a chance you'll reconsider :)

Sure, if someone comes up with a better way to improve scalability
for single-threaded applications.

> > 
> > > What is the user case?
> > 
> > Use case? Scalability. Pls go over Dror's presentation given at Sonoma -
> > he calls this SSQ.
>
> As far as I can tell he is talking about HW supported solution and not
> half baked SW one.

No, sharing a send queue must be done in software.  I don't really see the reason
for sarcasm: do you see value in sharing resources between multiple threads?
Why not multiple processes? Some people just don't want to program
in multithreaded environment.

> > 
> > > If multiple processes what to post to the same QP how will you
> > > ensure that right process will receive right completion event?
> > 
> > Same as with threads - memory for CQEs and locks will be allocated
> > in shared memory to make it possible for multiple processes to poll
> > CQ simultaneously, and they get completions in FCFS order.
> > What to do with them is up to the user.
>
> Are you going to use this API? How? There is no point in discussing user
> API without specifying HOW user will be using it. You have to ask what
> user want and design your API accordingly and not other way around.
> So suppose I want to use proposed API to implement super scalable MPI.

We'd come up with MPI_Send implementation inside libibverbs:). Think layered - I'd
like to make a minimal possible API change to make scalability improvements
possible.

> I setup shared QP/CQ/... and each rank start to post into the QP and
> receive completion from CQ and suppose rank A picked completion that
> belongs to rank B so I will need to setup out of band channel to pass
> this completion from A to B. This is not looks good at all to me.

This is not different from multiple threads sharing a CQ, really - and we do
support this already.  In the part of the message that you have cut out, I
showed some use cases that avoid this "side channel"
(which could be just shared memory btw).

-- 
MST