[ofa-general] Re: [PATCH RFC] libibumad: give up cpu time slice if write() falied

[Yevgeny Kliteynik]

Sasha Khapyorsky wrote:
> Hi Yevgeny,
> 
> On 17:26 Thu 29 Jan     , Yevgeny Kliteynik wrote:
>> While running opensm on a single-core CPU I've noticed the following problem:
>> when SA is stressed with many SA queries (such as when you run "osmtest -f f"
>> on a multi-core CPU machine), sometimes opensm fails to send responses.
>> It appears that send buffer gets full, and write() fails.
> 
> What is an errno?

write() returns -1, errno is EINVAL

>> Since the CPU has a single core, when the sender thread retries to send the
>> same packet, it fails again and again, because the driver didn't have the chance
>> to transmit something from the send buffer.
> 
> Why not? DMA doesn't require CPU cycles. And even if it is not DMA then
> an user space running thread will be rescheduled some days.
> 
>> Then I added 20 msec sleep after write() failure to make it give up the cpu time
>> slice, and I saw some improvement. When I added sleep of several seconds, the problem
>> disappeared completely, but then of course, the client's transaction will fail with
>> timeout unless you specifically increase the transaction timeout on the client side.
>>
>> So, what do you think about the following patch?
> 
> If errno is EAGAIN (and I would expect that it is) then you can handle
> this in an upper layer application (if it is relevant at all),

Since the errno is EINVAL, the application has no way knowing what
really happened. But probably the errno in this specific case should
be fixed - EAGAIN definitely makes more sense.

> and limiting library is not a good idea IMO.

OK, agree

-- Yevgeny

> For example how I would make
> stress-test then or benchmark? And what is the meaning of this "magic"
> number 20 ms?
> 
> Sasha
>