[openib-general] performance counters in /sys

[Grant Grundler]

On Fri, May 20, 2005 at 10:20:29AM -0600, Ronald G. Minnich wrote:
> yes, that is the accepted thing nowadays. And, boy, is it SLOW. SLOW,
> SLOW, SLOW.
> Open file, read ONE value, close file. Takes a long time. So we've seen 
> that for a number of values you really can't sample at interesting rates. 

Exactly. That's Mark's gripe too.

> Of course, you can speed it up a bit -- keep your files open, right?
> open once, read, fseek 0, read, ...
> Except then, if you need to unload the module, well, you can't: you have
> the /sys files open and the module can't really be unloaded.

That's good, IMHO. Unloading the HCA driver will disable the device.
On ia64 platforms (and probably PPC too) will lead to hard fail on the
next MMIO read since the device won't respond to MMIO space accesses.

"lsof /sys/class/infiniband" would help track down the monitoring tool.


> If I understood what to look for from the driver better (I lack knowledge 
> in this IB area :-) I could write a supermon kernel module for it.

The location/format of existing IB perf counters is advertised today.
See code in drivers/infiniband/core/sysfs.c that deals with:
	static struct attribute_group pma_group = {
		.name  = "counters",
		.attrs  = pma_attrs
	};

> Again, this only matters if you need performance, I'm not sure what
> people need here.

I talked with Mark Seger yesterday and he's looking for once per second
or subsecond reads - not Khz sample rates.  One answer might be to ignore
convention and add an "pma_all_attr".  For one or two cards, this should
be sufficient.

I think the right answer for Khz sample rates would be to add an interface
to a user space lib (e.g. libibverbs) that exports something like
"ethtool -d" uses to dump the NIC registers.

thanks,
grant