[ofa-general] uDAPL EVD queue length issue

[Jon Mason]

While working on OMPI udapl btl, I have noticed some "interesting"
behavior.  OFA udapl wants the evd queues to be a power of 2 and
then will subtract 1 for book keeping (ie, so that internal head and
tail pointers never touch except when the ring is empty).  OFA udapl
will report the queue length as this number (and not the original
size requested) when queried.  This becomes interesting when a power
of 2 is passed in and then queried.  For example, a requested queue
of length 256 will report a length of 255 when queried.  I cannot tell
if it is acceptable to get a size less than the one you request, based
on the udapl documentation.

Now during the setup of the ompi connection, it will try to make the
local parameters sufficient to run the programs.  Now if we try to
run a small amount of procs, then the defaults will be reset across
all nodes.  Since the defaults may not exactly match, udapl btl will
try to resize the queue (in this example 256 > 255).  When the call
finally makes it up to the ofa udapl code, it will bail because it
checks to see if the new size is less than or equal to the current
size + 1.

So if the ofa udapl code is working as designed, then the ompi udapl
btl code needs to have the proper boundary check for size + 1 (for
which I have a patch).  If not, then the ofa code need to be changed
to either round up to the next power of 2 if given a power of 2, or
return the size + 1 when queried.

So, which one is correct?

Thanks,
Jon

BTW, If anyone is interested, I have cut down dapltest to a very basic
test that will show this behavior 100% of the time.  I can make the
source available to whomever wants it.