[openib-general] Re: ibv_get_async_event

[Roland Dreier]

    Michael> If thats what we are trying to do, I'd like to propose
    Michael> another idea: when cq is destroyed, and after all cq
    Michael> events are queued to user, put a special "cq destroyed"
    Michael> event into the event queue.  When a user calls
    Michael> destroy_cq, and after releasing kernel/hardware
    Michael> resources, move the cq structure to a special cleanup
    Michael> list, userspace library will only destroy the userspace
    Michael> cq structure when it gets this event.

Makes sense.  I coded this up with the API as below:

	/**
	 * ibv_destroy_cq - Destroy a completion queue
	 * @cq: CQ to destroy.
	 *
	 * If ibv_req_notify_cq() has ever been called for @cq, then the
	 * CQ structure will not be freed until ibv_get_cq_event() has
	 * returned a "dead CQ" event (ie is_dead != 0) for this CQ.
	 *
	 * Calling ibv_destroy_cq() from one thread while another thread calls
	 * ibv_req_notify_cq() for the same CQ will lead to unpredictable
	 * results.  Don't do that.
	 */
	extern int ibv_destroy_cq(struct ibv_cq *cq);
	
	/**
	 * ibv_get_cq_event - Read next CQ event
	 * @context: Context to get CQ event for
	 * @comp_num: Index of completion event to check.  Must be >= 0 and
	 *   <= context->num_comp.
	 * @cq: Used to return pointer to CQ.
	 * @cq_context: Used to return consumer-supplied CQ context.
	 * @is_dead: If non-zero, then the event indicates that the CQ has
	 *   been destroyed and no more completion events will be generated
	 *   for the CQ.
	 */
	extern int ibv_get_cq_event(struct ibv_context *context, int comp_num,
				    struct ibv_cq **cq, void **cq_context,
				    int *is_dead);
	
I'll post the full kernel patches and userspace changes in separate
emails shortly.

    Michael> Instead of reference counting, we require the user to
    Michael> poll for this event after destroying the cq.  Since all
    Michael> events are polled from a single thread, when the library
    Michael> sees this event it knows the user does not keep a pointer
    Michael> to the cq any more.

Actually there's no rule that all events need to be read from a single
thread.  But if an application wants to use multiple threads and ends
up getting things out of order, then I think that's the app's fault.

 - R.