[ofa-general] Re: [PATCH RFC] sharing userspace IB objects

[Galen Shipman]

> Looking at the Dror's slides on slide 6 "Scalable Reliable  
> Connection" I
> see that wire protocol is extended to send DST SRQ as part of a  
> header.
> Receiver side then puts completion to appropriate CQ according this
> field. Have you proposition address this? How? Who will put this
> additional data on a wire (HW or libibverbs may be app)? Also I don't
> see this in Dror's slide, but completion of local operation should be
> demultiplexed to appropriate CQ too. WQE may contain additional field,
> for instance, that will tell where to put a completion. Once again who
> will do the demux in you proposition (HW, libiverbs or app)? The right
> answer is most certainly HW in both cases so will Hermon support this?
> Or may be you want to demultiplex everything inside libibvers? In this
> case I want to see design of this (preferably with performance  
> analysis).
>

While I think the SRC design makes sense I also have concerns  
regarding SSQ.
As Gleb has pointed out, if the hardware doesn't do the demux then  
the application has to.
It sounds like there are two proposals to deal with this hardware  
limitation in software (sigh).

1) Process A polls CQ, if WQE belongs to Process B, Process A will  
drop the WQE in a shared memory region that Process B  will poll.
      So we end up re-implementing shared memory completion semantics  
all over again for SSQ. I am concerned that there is both a latency  
hit (on average) and a memory scaling issue in that multiple QPs will  
now be replaced with shared memory completion fifos and a single SSQ QP.

2) Process A peeks CQ, if WQE belongs to Process B, it doesn't  
process it
	This has very bad implications for real applications, having to  
context switch to receive a WQE is bad

In my opinion the demux belongs in the hardware, otherwise we end up  
complicating an already complicated code base to support a feature  
which unless I am missing something will have no benefit to real  
applications.

- Galen



> --
> 			Gleb.
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/ 
> openib-general