Re: git: Bring in a simple event tracing library and POC utility

[Aggelos Economopoulos <aoiko@xxxxxxxxxxxxxx>]

Aggelos Economopoulos wrote:
> Aggelos Economopoulos wrote:
>> commit e7c0dbbaa9d5a2de52e8882628668615903e9132
>> Author: Aggelos Economopoulos <aoiko@cc.ece.ntua.gr>
>> Date:   Mon Feb 8 19:43:33 2010 +0200
>>
>>     Bring in a simple event tracing library and POC utility
[...]
> There's many things we could do with this infrastructure in place, maybe
> I should put together a mail outlining possible future directions.

One thing I'd like to add is a 'stats' (and maybe a 'plot') command. In
the past I've had to write a set of scripts to parse ktrdump output, to
do things such as calculate the time spent in an interrupt handler and
the used decriptors in a network interface tx ring. Now, adding
something like

evtranalyze -f /path stats evpair -b "begin intr" -e "end intr"

(where "begin intr" and "end intr" are the respective ktr format
strings; they could also be regular expressions) that produces a
mean/stddev is easy, but I'd like to go a bit further. Paired events are
a pattern that appears throughout the kernel and we could add
first-class support for it.

So what I want to do is use a standardized and trivial to parse format
string. For example,

"#compl:INT BEG DEV=%p"

where '#' is the special character that instructs libevtr to parse the
format string. In this case, libevtr would generate a special
EVTR_TYPE_COMPLETION event, with special fields:

struct evtr_event {
	...
	union {
		struct completion {
			const char *name;	/* "INT" */
			void *obj;		/* kernel pointer */
			/* or maybe const char *obj_name; see below */
		} compl;
		...
	}
}

With a bit of behind-the-scene magic we could provide the device name as
well (or maybe recording just the device name from the start would be
simpler, depends on whether we'd like to expose other device properties).

Then, one could use something like

evtranalyze stats duration "INT DEV=devname"
evtranalyze stats qlen "TXPKT NETIF=ifname"
evtranalyze stats qlen "DISKIO DEV=devname"

and the respective plot commands, e.g.

evtranalyze plothist duration "INT DEV=devname"

to create a histogram[0] of the distribution of interrupt times

or

evtranalyze plottime qlen "DISKIO DEV=devname"

to get a qlen-time graph (obviously one would need to make sure they
start with no pending completion events for the absolute numbers to make
sense, so I'm not sure if this approach makes a lot of sense. One could
just as well record "var:TXQLEN DEV=%p VAL=%d" events to trivially get a
reliable plot).

This is all a bit blue-sky, but I'm mainly throwing the idea out there
to see what people think of the approach. I think standardizing format
strings a bit is a good idea in any case.

These are the kind of features I started writing libevtr/evtranalyze for.

Aggelos

[0] or output that could be fed to a plotting program to generate said
histogram.