How to destroy IB resources (was Re: [ofa-general] Help - RDMA event files remain open after acknowledging them)

[Sean Hefty]

>I guess my question is, what's the best way to destroy IB resources? (Perhaps
>even, what's the best way to init them in the first place).

If you're destroying the CQ, there's no need to call ibv_get_cq_event() or
ibv_poll_cq(), unless you need completion information (for example, from flushed
receives).

However, every successful call to ibv_get_cq_event() needs a corresponding call
to ibv_ack_cq_event().  You can call ack(1) for each cq event, or count the
number of times that get returns success and call ack(get_cnt) once before
calling destroy.  Note that the count refers to the number of cq events, and not
the number of completions returned through ibv_poll_cq.

For your drain_cq() function, you should be safe doing something like this:

while (ibv_poll_cq(...) > 0)
	/* optional processing of any left over completions */;

ibv_ack_cq_event(...this_cqs_total_event_cnt); /* or ack after get */
ibv_destroy_cq(...);

>ibv_dealloc_pd(), ibv_destroy_cq() and ibv_destroy_comp_channel() all return
>error EBUSY

This sounds like a QP isn't being destroyed.  I'm not sure that anything else
fails CQ destruction with EBUSY.

Btw, if you're using the rdma_cm interface, then it's simpler to use the
rdma_create_qp/rdma_destroy_qp calls, which allows the rdma_cm to perform the QP
state transitions for you.

- Sean