[ofa-general] Re: [PATCH RFC] sharing userspace IB objects

[Gleb Natapov]

On Sun, Jul 01, 2007 at 01:05:30AM +0300, Michael S. Tsirkin wrote:
> > Quoting Roland Dreier <rdreier at cisco.com>:
> > Subject: Re: [PATCH RFC] sharing userspace IB objects
> > 
> >  > This is not directly related to SRC: this is an effort
> >  > to make it possible to share QPs, CQ etc across processes
> >  > in the same way as they can be currently shared across threads.
> >  > So assuming that we want multiple processes to post to
> >  > the same QP, how can we support this?
> > 
> > This looks like a lot of work for an unknown gain.  Who is going to
> > really use this?  ie is it worth the trouble?
> 
> I think Dror is the best person to answer this.
> Dror, could you please explain the need for shared send queue?
> 
SSQ is needed for scalability, no need to explain this (by the way RD
is needed for the same reason too. What's Mellanox plan to support it?
It is a part of Spec after all, so why to invent new shiny staff when it
is still possible to achieve better scalability without them).
We are discussing you implementation proposal and in my opinion it doesn't
fit application needs. I may be wrong here, so if there is somebody who
things that sending random completion to random processes it the best idea
ever and absence of this "feature" is the only thing that stops him from IB
adoption he may chime in here and voice his opinion.

Looking at the Dror's slides on slide 6 "Scalable Reliable Connection" I
see that wire protocol is extended to send DST SRQ as part of a header.
Receiver side then puts completion to appropriate CQ according this
field. Have you proposition address this? How? Who will put this
additional data on a wire (HW or libibverbs may be app)? Also I don't
see this in Dror's slide, but completion of local operation should be
demultiplexed to appropriate CQ too. WQE may contain additional field,
for instance, that will tell where to put a completion. Once again who
will do the demux in you proposition (HW, libiverbs or app)? The right
answer is most certainly HW in both cases so will Hermon support this?
Or may be you want to demultiplex everything inside libibvers? In this
case I want to see design of this (preferably with performance analysis).

--
			Gleb.