[ofa-general] uDAPL EVD queue length issue

[Arlin Davis]

Jon Mason wrote:
> While working on OMPI udapl btl, I have noticed some "interesting"
> behavior.  OFA udapl wants the evd queues to be a power of 2 and
> then will subtract 1 for book keeping (ie, so that internal head and
> tail pointers never touch except when the ring is empty).  OFA udapl
> will report the queue length as this number (and not the original
> size requested) when queried.  This becomes interesting when a power
> of 2 is passed in and then queried.  For example, a requested queue
> of length 256 will report a length of 255 when queried.  

Something is not right. You should ALWAYS get at least what you request. 
On my system with an mthca, a request of 256 gets you 511. It is the 
verbs provider that is rounding up, not uDAPL.

Here is my uDAPL debug output (DAPL_DBG_TYPE=0xffff) using dtest:

  cq_object_create: (0x519bb0,0x519d00)
dapls_ib_cq_alloc: evd 0x519bb0 cqlen=256
dapls_ib_cq_alloc: new_cq 0x519d60 cqlen=511

This is before and after the ibv_create_cq call. uDAPL builds it's EVD 
resources based on what is returned from this call.

I modified dtest to double check the dat_evd_query and I get the same:

8962 dto_rcv_evd created 0x519e80
8962 dto_req_evd QLEN - requested 256 and actual 511

What OFED release and device are you using?

-arlin