[ofa-general] IB performance stats (revisited)

[Jason Gunthorpe]

On Wed, Jun 27, 2007 at 01:48:04PM -0400, Hal Rosenstock wrote:

> Another approach would be to have the PMA inform the kernel that the
> counters were reset (perhaps including the values prior to the reset) so
> that these could be factored into the local set of counters. There is
> nothing in the spec that precludes this although it has not been
> implemented this way. Then there would't be a reason for a local manager
> to have to play these games. It would mean that there would need to be a
> performance manager running in the subnet which may not be acceptable
> for some installations; not sure.

If you are going to play those sorts of games I think it would better
to just effectively disable the PMA in the mellanox firmware and do
the following:

- The kernel periodically fetches the performance stats and aggregates
  them into a 64 wrapping counter. The kernel sends PMA mads into the
  mellanox firmware to read and reset the counters
- The new 64 bit stats are exported via sysfs/proc/whatever as
  wrapping counters
- When a PMA packet comes in the kernel services it rather than
  passing it on to the chip firmware.

Hopefully in future we could encourage new firmware/sillicon to
support exporting non-wrapping 64 bit counters to the OS so this ugly
mess wouldn't be needed.

FWIW, I agree with Mark that the current locally accessible counters
that are exactly the same as PMA mad values are virtually useless..

Jason