[ofa-general] uDAPL EVD queue length issue

[Steve Wise]

Jon Mason wrote:
> On Tue, Dec 04, 2007 at 11:40:17AM -0800, Arlin Davis wrote:
>> Jon Mason wrote:
>>> While working on OMPI udapl btl, I have noticed some "interesting"
>>> behavior.  OFA udapl wants the evd queues to be a power of 2 and
>>> then will subtract 1 for book keeping (ie, so that internal head and
>>> tail pointers never touch except when the ring is empty).  OFA udapl
>>> will report the queue length as this number (and not the original
>>> size requested) when queried.  This becomes interesting when a power
>>> of 2 is passed in and then queried.  For example, a requested queue
>>> of length 256 will report a length of 255 when queried.  
>> Something is not right. You should ALWAYS get at least what you request. On 
>> my system with an mthca, a request of 256 gets you 511. It is the verbs 
>> provider that is rounding up, not uDAPL.
>>
>> Here is my uDAPL debug output (DAPL_DBG_TYPE=0xffff) using dtest:
>>
>>  cq_object_create: (0x519bb0,0x519d00)
>> dapls_ib_cq_alloc: evd 0x519bb0 cqlen=256
>> dapls_ib_cq_alloc: new_cq 0x519d60 cqlen=511
>>
>> This is before and after the ibv_create_cq call. uDAPL builds it's EVD 
>> resources based on what is returned from this call.
>>
>> I modified dtest to double check the dat_evd_query and I get the same:
>>
>> 8962 dto_rcv_evd created 0x519e80
>> 8962 dto_req_evd QLEN - requested 256 and actual 511
>>
>> What OFED release and device are you using?
> 
> I'm running OFED 1.2.5 and using Chelsio.
> 
> The behavior of the iwch_create_cq in
> drivers/infiniband/hw/cxgb3/iwch_provider.c is to allocate the amount
> given (rounded to the power of 2).  So this function will give 256 if
> 256 is requested, but uDAPL will consume one of those for book keeping
> and thus only have 255.
> 
> For my clarification, the provider should take into account the
> bookkeeping of uDAPL and roundup to the next power of 2 when given a
> power of 2 size?  I'm probably being thick, but why doesn't uDAPL
> increase the size requested by one before passing the request to the
> provider (or is this the documented behavior of the function and the
> provider should conform)?
> 
> Thanks,
> Jon

 From the linux rdma verbs perspective, ibv_create_cq() will create a cq 
that is >= the requested depth.  The fact that mthca always bumps the 
size up to the next power of 2 isn't something udapl can rely on.

Here's the crux:  If creating a udapl evd of 256 results in a cq of 256 
and the udapl returns a evd with size 255, then udapl is broken...

My 2 cents...

Stevo.