[openib-general] Patch for review: ipath mmaped CQs, QPs, SRQs [1 of 2]

[Ralph Campbell]

On Fri, 2006-06-16 at 09:25 -0700, Roland Dreier wrote:
>  > +	/* Unmap the old queue so we can resize it. */
>  > +	size = sizeof(struct ipath_cq_wc) +
>  > +		(sizeof(struct ipath_wc) * cq->ibv_cq.cqe);
>  > +	(void) munmap(cq->queue, size);
>  > +	ret = ibv_cmd_resize_cq_resp(ibcq, cqe, &cmd, sizeof cmd,
>  > +				     &resp.ibv_resp, sizeof resp);
>  > +	if (ret) {
>  > +		pthread_spin_unlock(&cq->lock);
>  > +		return ret;
>  > +	}
> 
> It seems that this method of returning a new buffer address to mmap
> from the resize operation leads to some really nasty error handling
> though.  If the resize operation fails (either because of bad
> userspace values or because the kernel is out of memory and can't
> allocate a new buffer) then the old CQ is gone, possibly beyond recovery.
> 
> mthca avoids all this by allocating a resize buffer in advance.
> 
>  - R.

I agree. The kernel driver is careful to allocate the new queue
and copy the old contents to the new one atomically. The issue
is making sure the old queue isn't still being used by some
other thread. I guess if the semantics for resize are
that an error means the old mmap is still valid but
if the resize succeeds, the old mmap is invalid, then there isn't
an error recovery issue. All user level threads lock before
using the queue address so the change of address is protected.


-- 
Ralph Campbell <ralphc at pathscale.com>