[ofa-general] Re: Re: [PATCH RFC] sharing userspace IB objects

[Gleb Natapov]

On Tue, Jun 26, 2007 at 05:02:39PM +0300, Michael S. Tsirkin wrote:
> > Quoting Gleb Natapov <glebn at voltaire.com>:
> > Subject: Re: Re: [PATCH RFC] sharing userspace IB objects
> > 
> > On Tue, Jun 26, 2007 at 03:58:02PM +0300, Michael S. Tsirkin wrote:
> > > > > No, sharing a send queue must be done in software.  I don't really see the reason
> > > > > for sarcasm: do you see value in sharing resources between multiple threads?
> > > > > Why not multiple processes? Some people just don't want to program
> > > > > in multithreaded environment.
> > > >
> > > > Yes I see the value in sharing resources between threads and processes
> > > > if done right. This proposition is far from being right.
> > > 
> > > Ahem, *what* are you talking about? Sharing resources between threads was supported in
> > > libibverbs 1.0, *right from the start*. This is still the case with 1.1, and this API
> > > matches verbs quite closely which means that it can work pretty much on any
> > > hardware.
> > 
> > Why do you think that I have a problem with multithreaded application is
> > beyond my understanding. I have a problem with you thinking that peaking a
> > completion by random process in FCFS order is a good idea.
> 
> Should that have been "picking"?  I keep telling you. With multithreaded
Yes "picking". Sorry :)

> applications *that's what currently happens*. If multiple threads poll a CQ,
> which one gets which completion is currently unspecified. Are you
> worried about this? If not, why are you worried when multiple
> processes do this?
You've missed my sentence about difference between multithreaded
application and what you propose. The difference is HUGE (I can't write
bigger letters sorry about that). I can design a multithreaded MPI so
that each thread will be capable to progress MPI send/recv request (and then
I don't care what thread gets which completion. I can't do it with multiprocess
scenario.

> 
> Look here, hardware features do *not* just materialize when you build an API for
> them.  What good would a pretty API that no hardware supports be?  It's the
> other way around: I'm trying to extend our API to improve scalability with
> existing hardware.
> 
Then this API will stick forever. And HW implementation will have
different API anyway. And that what I am trying to point.
I don't thing Mellanox implemented SRQ API before it was available in
HW. If Mellanox think this is such a great idea (and it is) why not put
implementation where it belongs (in HW that is).

--
			Gleb.