[openib-general] cq polling order

[Dotan Barak]

On Monday 07 August 2006 11:14, Devesh Sharma wrote:
> Hi, Dotan Thanks for quick reply.
> 
> On 8/7/06, Dotan Barak <dotanb at mellanox.co.il> wrote:
> >
> > Hi.
> >
> > On Monday 07 August 2006 10:06, Devesh Sharma wro
> > > Hello everybody,
> > > I have a query regarding cq poll concept.
> > > Consider the following situation:
> > >
> > > Consumer has posted 2 SEND operations after that it posted 1 RDMA_READ
> > > operation and againg 1 SEND operation ( Not posted with Barrier Fence,
> > Is it
> > > expected from consumer that it can post without Barrier Fence after RDMA
> > > READ? ), now while polling in what order completions should be returned
> > by
> > > verbs?
> > >
> > > Is it expected by consumer that completions will be polled in posting
> > order
> > > or they can come out of order?
> > > Polling order 2 SEND_COMP, 1 RDMA_READ_COMP, 1 SEND_COMP
> > >  OR
> > > Polling order 3 SEND_COMP, 1 RDMA_READ_COMP
> > >
> > > which is expected?
> >
> > The order the completions must be the same as the order of the WR that you
> > posted to the Send Queue.
> > so, you should expect the first option.
> 
> 
> But RDMA Read  may complete out of order. Is it means that HCA Driver should
> be implemented such that order is maintained in such situations?
> 
In the IB spec 10.8.5:  "... Work Completions are always returned in the order submitted
to a given Work Queue with respect to other Work Requests on that Work Queue."
so, the order of the completions in the CQ should be the same order of the posts WR
on that Wort Queue. 

> as you mentioned, if the fence bit is enable on a specific SR, the HCA need
> > to wait until all the previous SR
> > will be finished (according to the fence laws in the IB spec ..).
> 
> 
> If consumer is specifing Fence then there is No problem of polling order,
> Since, HCA will not start processing next WR untill all WR prior to this WR
> completes, Hence polling will inherently be done in order, But issue is if
> Cosumer dose not uses FENCE flag in next send operation after RDMA READ.
I believe that the order of the completions in the CQ is not being effected by the 
FENCE bit. The FENCE bit will effect the processing timing:
if one will post RDMA Read and then Send With/Without FENCE and he will use the 
same local memory addresses, the difference that he will see is in the data (there is a race
between the RDMA Read and the Send: if the Send will start after the Read was finished or not).


Dotan