[ofa-general] uDAPL EVD queue length issue

[Steve Wise]

Arlin Davis wrote:
>>>
>>> I'm running OFED 1.2.5 and using Chelsio.
>>>
>>  From the linux rdma verbs perspective, ibv_create_cq() will create a 
>> cq that is >= the requested depth.  The fact that mthca always bumps 
>> the size up to the next power of 2 isn't something udapl can rely on.
> 
> It doesn't.
> 
> uDAPL passes the users requested qlen directly to the verbs 
> ibv_create_cq call (dapl_openib_cq.c) and then uses the returned qlen to 
> allocate EVD's (dapl_evd_util.c) and a ring buffer 
> (dapl_ring_buffer_util.c) for managing the free and pending events.
> 
> The EVD's are allocated based on returned qlen from verbs and the 
> ring_buffer is allocated based on qlen, next power of 2 minus 1. Unless 
> I am missing something, I don't see how we can get less then what is 
> requested.
> 
>>
>> Here's the crux:  If creating a udapl evd of 256 results in a cq of 
>> 256 and the udapl returns a evd with size 255, then udapl is broken...
> 
> Yes, I agree. But I don't see how this is happening.
> 
> Here is output from my dtest when requesting 256 and verbs returning the 
> same qlen. You can see before and after verbs we get 256, the rbufs are 
> 511, and the query EVD call returns 256 to the user.
> 
>  cq_object_create: (0x519bb0,0x519d00)
> dapls_ib_cq_alloc: evd 0x519bb0 cqlen=256
> dapls_ib_cq_alloc: new_cq 0x519d60 cqlen=256
>  setup_async_cb: ia 0x518270 type 2 hdl 0x519bb0 cb 0x2a957c8e70 ctx 
> 0x519bb0
>  >>> rbuf_alloc: size 256 rsize 511
>  >>> rbuf_alloc: size 256 rsize 511
> dapl_evd_create () returns 0x0
> 
> 9920 dto_req_evd QLEN - requested 256 and actual 256
> 9920 Create events done
> 
> Can you turn up DAPL debug(DAPL_DBG_TYPE=0xffff) so I can see what is 
> happening?
> 
> Thanks,
> 
> -arlin
> 
> 
> 
> 

Sorry for the noise guys.  Looks like this is a cxgb3 bug.


Steve.