[openib-general] Re: performance counters in /sys

[Ronald G. Minnich]

On Mon, 23 May 2005, Michael S. Tsirkin wrote:

> > I guess the thing that has me mystified about all this is I can 
> > certainly appreciate the potential 'goodness' of having 1 var/file for 
> > user oriented access but perhaps one of the better examples of why this 
> > is just a bad idea for programmatic access is the individual process 
> > stats.  Is the implication of this that some day those too would be 
> > moved to /sys as one stat per file?!?  Can you imaging trying to run top 
> > or ps if that were to happen?
> Does someone care enough to post a benchmark?

I have one I will try to post, but ... I did create a test program to see
if I could run supermon in user mode with no kernel module. Supermon
kernel module presents one file in /proc/sys that provides an s-expression
containing info on avenrun, enet stats, disk, etc.  There is an additional
file presnted with an s-expression containing descriptors of what the data
means, whether it is bounded, and if bounded, what the min/max is. The
module works correctly in the presence of hotplug (s-expressions are your
friend).

On Linux, we can hand out S-expressions at 12 Khz; on Plan 9, at 30 Khz --
i.e., 12,000 and 30.000 samples/second.  Obviously, you don't run that
fast, the reason to know the max is to give you an idea of the impact of
reading the file at, e.g., 100hz; in the case of supermon, the impact is
in essence zero.

My simple test program opened the files that the supermon kernel module
provides, and on a request for data, did a read() and lseek(0),
concatenating the 'single value per file' into an s-expression.

Performance was pretty bad, I figure that you really can't sample at more 
than 1/50 hz (once every 50 seconds) without impacting the cpu performance 
in a measurable way. Plus, keeeping the files open means you can't unload 
modules. 

So, I went to open/read/close for each file, to allow module unloading to
work, and of course performance was even worse.

One value per file is a  fine rule if you don't need high performance data 
collection. In supermon, we can sample at 100hz with no impact on the 
apps, and have actually observed mpi collectives in progress in that way. 
It's pretty neat. 

One stat per file is not a good idea if you need low-impact, high 
performance data collection.

ron