How to destroy IB resources (was Re: [ofa-general] Help - RDMA event files remain open after acknowledging them)

[Nitin Mehrotra]

Hmm, I do amortize the cost of  ibv_ack_cq_event() over multiple 
ibv_get_cq_event() calls; however when the shutdown is in progress I 
don't have the last event that was "gotten" so I have to call 
ibv_get_cq_event() one last time to get an event to acknowledge against. 
I suppose it's probably better to keep the last event processed if it 
hasn't been acknowledged and use it to issue the final acknowledge when 
shutting down. Then I wouldn't have to make that ibv_get_cq_event() call.

One last question, when I create the completing event queue I set it to 
non-blocking but I find that during shutdown I have to do that again 
before making the final call to ibv_get_cq_event() otherwise it blocks. 
Which I suppose is why it returns EAGAIN when there are no pending 
events, but I don't understand why I have to set it to non-blocking again.

Anyway, much thanks for all your help.

Nitin

Sean Hefty wrote:
>> I guess my question is, what's the best way to destroy IB resources? (Perhaps
>> even, what's the best way to init them in the first place).
>>     
>
> If you're destroying the CQ, there's no need to call ibv_get_cq_event() or
> ibv_poll_cq(), unless you need completion information (for example, from flushed
> receives).
>
> However, every successful call to ibv_get_cq_event() needs a corresponding call
> to ibv_ack_cq_event().  You can call ack(1) for each cq event, or count the
> number of times that get returns success and call ack(get_cnt) once before
> calling destroy.  Note that the count refers to the number of cq events, and not
> the number of completions returned through ibv_poll_cq.
>
> For your drain_cq() function, you should be safe doing something like this:
>
> while (ibv_poll_cq(...) > 0)
> 	/* optional processing of any left over completions */;
>
> ibv_ack_cq_event(...this_cqs_total_event_cnt); /* or ack after get */
> ibv_destroy_cq(...);
>
>   
>> ibv_dealloc_pd(), ibv_destroy_cq() and ibv_destroy_comp_channel() all return
>> error EBUSY
>>     
>
> This sounds like a QP isn't being destroyed.  I'm not sure that anything else
> fails CQ destruction with EBUSY.
>
> Btw, if you're using the rdma_cm interface, then it's simpler to use the
> rdma_create_qp/rdma_destroy_qp calls, which allows the rdma_cm to perform the QP
> state transitions for you.
>
> - Sean
>   
> ------------------------------------------------------------------------
>
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com 
> Version: 8.5.392 / Virus Database: 270.13.56/2302 - Release Date: 08/14/09 06:10:00
>
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20090814/d7a4ad88/attachment.html>