[openib-general] Re: [PATCH 2/6] [RFC] mthca kernel changes for resizeCQ

[Michael S. Tsirkin]

Quoting r. Roland Dreier <rdreier at cisco.com>:
> Subject: Re: [PATCH 2/6] [RFC] mthca kernel changes for resizeCQ
> 
>     Michael> I think I see a problem with this approach: if a ULP
>     Michael> performs CQ poll while mthca_RESIZE_CQ is in progress, it
>     Michael> might get a false indication that the CQ is empty since
>     Michael> CQEs are being written to the new buffer already.
> 
> I tried to handle this in the poll CQ operation:

Right, sorry, I missed this one.


> 	int mthca_poll_cq(struct ib_cq *ibcq, int num_entries,
> 			  struct ib_wc *entry)
> 	
> 		/*...*/
> 	
> 		/*
> 		 * If a CQ resize is in progress and we discovered that the
> 		 * old buffer is empty, then peek in the new buffer, and if
> 		 * it's not empty, switch to the new buffer and continue
> 		 * polling there.
> 		 */
> 		if (unlikely(cq->resize_buf				&&
> 			     cq->resize_buf->state == CQ_RESIZE_READY	&&
> 			     err == -EAGAIN)) {

Might be a good idea to test err == -EAGAIN first: err is likely a register,
and once you get a valid CQE returning it to ULP ASAP has direct latency impact.

> and so on...
> 
> I'm not sure I got it right but it looks OK to me.
> 
> I don't really see any other way to do this, because the RESIZE_CQ
> command can block, so we can't lock out poll CQ operations while the
> firmware command is executing.

I agree, neither do I. Did you have a chance to profile this code?
Is there any performance impact on IPoIB?


-- 
Michael S. Tsirkin
Staff Engineer, Mellanox Technologies