[ofw] Any interrupt coalescing in WinOF?

[Chris Worley]

On Tue, Oct 12, 2010 at 2:01 PM, Tzachi Dar <tzachid at mellanox.co.il> wrote:
> Do you believe that interupt moderation will be able to help?

Yes.  The CPU's pegged seem to be the current bottleneck.  If that
load could be distributed to other CPU's or more work done per
interrupt, then that should help.  Of course, I don't understand
what's going on; I assume that the HCA can buffer the work... if
coalescing just backs-up the pipeline, then it will not help.
>
> There is currently no way for the user can control that but I hope that this is a small change in the code.
>
> By the way, are you running connect2 or connect?

I believe these are ConnectX QDR HCA's, as neither are brand new...
but I think I have access to newer HCA's if that would help.

Thanks,

Chris
>
> Thanks
> Tzachi
>
>> -----Original Message-----
>> From: Chris Worley [mailto:worleys at gmail.com]
>> Sent: Tuesday, October 12, 2010 7:10 PM
>> To: Tzachi Dar
>> Cc: ofw at lists.openfabrics.org
>> Subject: Re: [ofw] Any interrupt coalescing in WinOF?
>>
>> On Tue, Oct 12, 2010 at 10:54 AM, Tzachi Dar <tzachid at mellanox.co.il>
>> wrote:
>> > There is no interrupt coalescing on winof although the hw does
>> support that (I assume that you are running connectx).
>> >
>> > You can verify if the number of interrupts is big by running perfmon.
>> Select the cpu, and choose number of interrupts per second.
>> >
>> > Look at this when your test is running and when it is not.
>>
>> Although there are some Heisenberg issues w/ this test (perfmon locks
>> up trying to measure cores that are pegged), I was able to confirm
>> that all the interrupts are hitting these two cores, and assume each
>> core is getting the interrupts for one HCA.
>>
>> Thaks,
>>
>> Chris
>> >
>> > Thanks
>> > Tzachi
>> >
>> >> -----Original Message-----
>> >> From: Chris Worley [mailto:worleys at gmail.com]
>> >> Sent: Tuesday, October 12, 2010 6:49 PM
>> >> To: Tzachi Dar
>> >> Cc: ofw at lists.openfabrics.org
>> >> Subject: Re: [ofw] Any interrupt coalescing in WinOF?
>> >>
>> >> On Tue, Oct 12, 2010 at 10:34 AM, Tzachi Dar
>> <tzachid at mellanox.co.il>
>> >> wrote:
>> >> > What test/ulp are you using?
>> >>
>> >> IOMeter with one worker atop each of 4 SRP-imported/multipathed
>> >> (round-robin) drives and a queue depth of 64.  I'm seeing ~450K
>> random
>> >> 4KB write IOPS.
>> >>
>> >> With two HCA's, both running the latest firmware (as of Friday), I
>> see
>> >> two cores on one NUMA node totally pegged/consumed, and affinity is
>> >> set on the Dynamo's to avoid these cores... so something is either
>> >> polling or interrupt handling on these.
>> >>
>> >> I'm running the latest 2.3RC4 WinOF.
>> >>
>> >> Thanks,
>> >>
>> >> Chris
>> >> >
>> >> > Thanks
>> >> > Tzachi
>> >> >
>> >> >> -----Original Message-----
>> >> >> From: ofw-bounces at lists.openfabrics.org [mailto:ofw-
>> >> >> bounces at lists.openfabrics.org] On Behalf Of Chris Worley
>> >> >> Sent: Tuesday, October 12, 2010 6:33 PM
>> >> >> To: ofw at lists.openfabrics.org
>> >> >> Subject: [ofw] Any interrupt coalescing in WinOF?
>> >> >>
>> >> >> I'm seeing the interrupt handlers completely consuming a CPU and
>> >> >> becoming a bottleneck, and no interrupt load balancing across
>> CPU's
>> >> >> (except w/ multiple HCA's, each HCA consumes one CPU w/
>> interrupts).
>> >> >>
>> >> >> Is interrupt coalescing manually tunable in WinOF?
>> >> >>
>> >> >> Or, am I totally off-base, and seeing polling and not interrupt
>> >> >> behavior?
>> >> >>
>> >> >> Thanks,
>> >> >>
>> >> >> Chris
>> >> >> _______________________________________________
>> >> >> ofw mailing list
>> >> >> ofw at lists.openfabrics.org
>> >> >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
>> >> >
>> >
>