[ofw] Adding Performance Counters support for WinOF

[Moshe Haim]

Hi,

Took me a while :)

I repeated the experiment in Windows and in Linux and I was mistaken.
Both in Linux and windows the counters report the amount of transferred bytes / 4.
This basically means that I'm having the same issue with counters filling up in Linux as in windows - which make sense since we have only 32-bits.

Now I have another question. There is an additional counter "set" called Recv/XmitPkts, which obviously grows slower than number of bytes.
And this caused me to think - If I have the packet size and the counter in packets I can get to a much higher limit of transferred bytes till the counter hits 32-bit.

1. Is the pkt size constant per implementation - for instance let's say 512 bytes in Windows (WinOF) and some other value in OFED for Linux?
2. Can I get the packet size through some API?

Thanks,
Moshe.

-----Original Message-----
From: Hal Rosenstock [mailto:hal.rosenstock at gmail.com] 
Sent: Tuesday, February 24, 2009 3:43 PM
To: Moshe Haim
Cc: ofw at lists.openfabrics.org
Subject: Re: [ofw] Adding Performance Counters support for WinOF

On Tue, Feb 24, 2009 at 2:08 AM, Moshe Haim <Moshe-ha at orbotech.com> wrote:
> On the Linux system I cleared the counters and used perfquery to see
> that they are 0.
> Then I ran a program I wrote that transfers 1GB of data.
> When I ran perfquery again, I then calculated 1GB / received counter and
> got to a relation of around 100MB for each increase in the counter.

Are you saying the RcvData counter is 10 ?

> Doing the equivalent in Windows indeed brought me to the 4 bytes
> increase (as I assumed it would be).

I'm not following what you mean by this. Are you saying it's the
proper/expected number accounting for data being in 4 byte units ?

-- Hal

> -----Original Message-----
> From: Hal Rosenstock [mailto:hal.rosenstock at gmail.com]
> Sent: Wednesday, February 11, 2009 6:14 PM
> To: Moshe Haim
> Cc: ofw at lists.openfabrics.org
> Subject: Re: [ofw] Adding Performance Counters support for WinOF
>
> On Sun, Feb 8, 2009 at 1:54 AM, Moshe Haim <Moshe-ha at orbotech.com>
> wrote:
>> The result of running perfquery -e:
>> ibwarn: [20545] main: PerfMgt ClassPortInfo 0x0 extended counters not
>> indicated
>>
>> perfquery: iberror: failed: perfextquery
>
> Yes, the extended 64 bit counters are not supported on that device type
> :-(
>
> No clue as to the data counter anomaly you mentioned: "increments are
> around 1MB - so if my counter shows 0x64 it means ~100MB of data were
> transferred." The low level counter is bytes * 4. Is this that counter
> or some other aggregated counter in software ? Is some performance
> manager running ? Are these counters being reset ?
>
> -- Hal
>
>> -----Original Message-----
>> From: Hal Rosenstock [mailto:hal.rosenstock at gmail.com]
>> Sent: Thursday, February 05, 2009 2:23 PM
>> To: Moshe Haim
>> Cc: ofw at lists.openfabrics.org
>> Subject: Re: [ofw] Adding Performance Counters support for WinOF
>>
>> On Thu, Feb 5, 2009 at 1:47 AM, Moshe Haim <Moshe-ha at orbotech.com>
>> wrote:
>>> I am using CentOS-5 with kernel 2.6.18-8.el5.
>>> Using ofed_info the OFED version is: OFED-1.2.5.5
>>>
>>> From what I understood in previous E-Mails 64-bit counters are not
>>> supported by the fw. And when I run perfquery I see that the counters
>>> are 32-bit long.
>>
>> Would you try perfquery -e for the different devices that you want the
>> counters ?
>>
>> -- Hal
>>
>>> Thanks,
>>> Moshe.
>>>
>>> -----Original Message-----
>>> From: Hal Rosenstock [mailto:hal.rosenstock at gmail.com]
>>> Sent: Monday, February 02, 2009 6:31 PM
>>> To: Moshe Haim
>>> Cc: Sean Hefty; ofw at lists.openfabrics.org
>>> Subject: Re: [ofw] Adding Performance Counters support for WinOF
>>>
>>> On Mon, Feb 2, 2009 at 10:13 AM, Moshe Haim <Moshe-ha at orbotech.com>
>>> wrote:
>>>> So it seems that with the new libraries I will be able to access the
>>>> information directly (through API) - which will be easier for me.
>>>>
>>>> However, the issue of the counters filling up so fast remains as is.
>>>
>>> Are the extended (64 bit) counters supported ? If so, those won't
> fill
>>> up so quickly.
>>>
>>>> I have made a few additional checks and updated my firmware. The
>>>> counters on windows are counted in 4 bytes increments. I.e if my
>>> counter
>>>> shows 0x64 then it means that 100*4 (400) bytes were transferred.
>>>
>>> The data counters are spec'd in bytes*4.
>>>
>>>> However, when I checked the Linux system using perfquery I am
> getting
>>> a
>>>> totally different ratio.
>>>> It appears that the counters there are still 32-bit wide, but the
>>>> increments are around 1MB - so if my counter shows 0x64 it means
>>> ~100MB
>>>> of data were transferred.
>>>
>>> Are you sure ? What Linux version is being used ?
>>>
>>> -- Hal
>>>
>>>> Can someone please tell me:
>>>> 1. Where does this difference come from?
>>>> 2. Since we are talking about 10Gbit (and more) transfer rates, why
>> do
>>>> the counters count in (4x)bytes? Since most chances they will fill
> up
>>>> very fast (as seen)
>>>>
>>>> Thanks,
>>>> Moshe.
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Sean Hefty [mailto:sean.hefty at intel.com]
>>>> Sent: Thursday, January 15, 2009 8:33 PM
>>>> To: Moshe Haim; ofw at lists.openfabrics.org
>>>> Subject: RE: [ofw] Adding Performance Counters support for WinOF
>>>>
>>>>>I further discussed it with my manager who worked with the Linux
>>>> version
>>>>>and I was told that on Linux there was a problem with perfquery
> since
>>>> it
>>>>>was neither thread nor process safe - i.e. calling it concurrently
>>> from
>>>>>2 or more threads/processes would crash.
>>>>>In addition, I can only assume perfquery is an executable.
>>>>
>>>> perfquery is an executable.  I'm not sure what's meant about being
>>>> thread or
>>>> process safe, since it's a stand alone program.
>>>>
>>>>>I also assume that perfquery uses one of the more "lower levels"
> dlls
>>>>>mentioned below in order to get the performance counter information
>>>>>(libibumad?).
>>>>
>>>> perfquery sends MADs that query the performance counters.  The
>>> counters
>>>> are
>>>> returned in a MAD.  The MAD format and behavior is defined by the IB
>>>> spec.
>>>>
>>>>>To be more concrete, if I'd want to directly access the counter
>>>>>information I'll simply need to link my project against that DLL and
>>>>>call an API method - is that assumption correct?
>>>>
>>>> You can send and process your own MADs.  In this case, you can write
>>> to
>>>> any of
>>>> MAD libraries.
>>>>
>>>> - Sean
>>>>
>>>> _______________________________________________
>>>> ofw mailing list
>>>> ofw at lists.openfabrics.org
>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
>>>>
>>>
>>
>