[openib-general] Re: [PATCH 2/6] [RFC] mthca kernel changes for resizeCQ

[Caitlin Bestler]

openib-general-bounces at openib.org wrote:
> Quoting r. Roland Dreier <rdreier at cisco.com>:
>> I'm not sure I got it right but it looks OK to me.
>> 
>> I don't really see any other way to do this, because the RESIZE_CQ
>> command can block, so we can't lock out poll CQ operations while the
>> firmware command is executing.
> 
> I thought about this some more. Its unfortunate that there's
> an overhead on each poll cq operation.
> 
> We could avoid this overhead by:
> - adding poll_cq callback and lock into the struct ib_cq,
>   initializing them on cq creation.
> - modifying ib_poll_cq to
> 
> static inline int ib_poll_cq(struct ib_cq *cq, int num_entries,
>                              struct ib_wc *wc) {
> 	unsigned long flags;
> 	int ret;
> 	spin_lock_irqsave(cq->lock, flags);
>         ret = cq->poll_cq(cq, num_entries, wc);
> 	spin_unlock_irqrestore(cq->lock, flags);
> 	return ret;
> }
> 
> Now, device driver can modify the poll_cq callback in the cq
> to a slower version that checks two buffers for the duration of the
> resize operation. 
> 
> How does this sound?

You can get the overhead even lower by just switching to a new
cq ring when you hit the end of the current one. But even that
sounds far too expensive for any device that does not provide
a true benefit for resizing the CQ.

For those devices the best support is provided when the consumer
allocates the maximum, any logical resizing to a smaller size is
ignored, and there is *no* extra checking (let alone a spinlock)
for each cq poll.