[openib-general] Re: performance counters in /sys

[Michael S. Tsirkin]

Quoting r. Ronald G. Minnich <rminnich at lanl.gov>:
> Subject: Re: performance counters in /sys
> 
> 
> 
> On Mon, 23 May 2005, Michael S. Tsirkin wrote:
> 
> > > I guess the thing that has me mystified about all this is I can 
> > > certainly appreciate the potential 'goodness' of having 1 var/file for 
> > > user oriented access but perhaps one of the better examples of why this 
> > > is just a bad idea for programmatic access is the individual process 
> > > stats.  Is the implication of this that some day those too would be 
> > > moved to /sys as one stat per file?!?  Can you imaging trying to run top 
> > > or ps if that were to happen?
> >
> > Does someone care enough to post a benchmark?
> 
> I have one I will try to post

Good, it shall then be possible to profile it and look for a solution.

> But ... I did create a test program to see
> if I could run supermon in user mode with no kernel module. Supermon
> kernel module presents one file in /proc/sys that provides an s-expression
> containing info on avenrun, enet stats, disk, etc.  There is an additional
> file presnted with an s-expression containing descriptors of what the data
> means, whether it is bounded, and if bounded, what the min/max is. The
> module works correctly in the presence of hotplug (s-expressions are your
> friend).
> 
> On Linux, we can hand out S-expressions at 12 Khz; on Plan 9, at 30 Khz --
> i.e., 12,000 and 30.000 samples/second.  Obviously, you don't run that
> fast, the reason to know the max is to give you an idea of the impact of
> reading the file at, e.g., 100hz; in the case of supermon, the impact is
> in essence zero.
> 
> My simple test program opened the files that the supermon kernel module
> provides, and on a request for data, did a read() and lseek(0),

Did you try pread(2)?

> concatenating the 'single value per file' into an s-expression.
>
> Performance was pretty bad, I figure that you really can't sample at more 
> than 1/50 hz (once every 50 seconds) without impacting the cpu performance 
> in a measurable way. Plus, keeeping the files open means you can't unload 
> modules. 
> 
> So, I went to open/read/close for each file, to allow module unloading to
> work, and of course performance was even worse.

By how much?

> One value per file is a  fine rule if you don't need high performance data 
> collection. In supermon, we can sample at 100hz with no impact on the 
> apps, and have actually observed mpi collectives in progress in that way. 
> It's pretty neat. 
> 
> One stat per file is not a good idea if you need low-impact, high 
> performance data collection.
> 
> ron
> 

Its not immediately obvious from the description
above that its the "one value per file" that is the performance bottleneck,
is it?

For example, I'm not familiar with the s-expression format.
Is it possible the extra reformatting/string copying from the
read buffuer to s-expression eats the CPU?

Well, I guess when we see the benchmark it shall be possible
to judge where does the time go. If all of sysfs has a speed problem,
the right way seems to solve it and not to add more /proc files
as a work-around, as supermon module seems to do.

thanks,

-- 
MST - Michael S. Tsirkin