Recorders

The recording module consists of the following data collectors:

Item	Description	Based On	FreeBSD	RedHat	(Open)Solaris
sysrec	system utilisation and queuing	Perl5, Kstat	n/a	n/a	yes
cpurec	per-cpu detailed statistics	Perl5, Kstat	n/a	n/a	yes
nicrec	network interface statistics	Perl5, Kstat	n/a	yes	yes
netrec	network protocol statistics	Perl5, netstat	n/a	n/a	yes
zonerec	zone statistics	Ksh, prstat	n/a	n/a	yes
corerec	CMT T1, T2 processor statistics	Ksh, Perl5, cpustat	n/a	n/a	yes
jvmrec	garbage collection statistics	Ksh, jstat	yes	yes	yes
webrec	HTTP response time recorder	Java, Apache HTTPClient	yes	yes	yes

sysrec

sysrec is a utility, part of K9Toolkit, author Brendan Gregg. The toolkit is a collection of free Perl scripts used to troubleshoot and observe (Open)Solaris systems. Check Appendix for more details. The recorder has been modified to output its data into RRD format. sysrec records system utilisation and certain additional kernel statistics used as a starting point in observing the system's general health. The output from sysrec is displayed below:

timestamp	CPU Util %	Mem Util %	Disk Util %	Net Util %	Run Queue	Scan rate	Disk waits	Net errors
1225038537:	10.56:	70.79:	9.87:	0.13:	0.02:	0.02:	0.19:	0.00
1225038539:	3.92:	70.79:	0.00:	0.00:	0.00:	0.00:	0.00:	0.00
1225038538:	2.94:	70.79:	0.00:	0.00:	0.00:	0.00:	0.00:	0.00

cpurec

cpurec is a utility, collecting per-CPU data from kstat The recorder outputs its data under RRD format. cpurec used mainly to observe CPU activity and analyse how the CPUs are used in the system. Useful for capacity planning. Recording points:

timestamp: seconds since Epoch
Xcalls: rate of multiprocessor cross calls
Intr: rate of intrerrupts
iThr: rate of interrupts threads
Csw: rate of context switches
Icsw: rate of involuntary context switches
Migr: rate of migrations
Smtx: rate of kernel mutexes
Syscalls: rate of system calls
User: percentage of time spent in user mode
Sys: percentage of time spent in sys mode
Idle: percentage of time spent in idle mode

The output from cpurec is displayed below:

timestamp

Cpuid

Xcalls

Intr

iThr

Csw

Icsw

Migr

Smtx

Syscalls

User %

Sys %

Idle %

1225039500:

97:

568:

49:

1195:

69:

190:

40:

4183:

5.95:

4.37:

89.68

1225039500:

98:

936:

513:

1169:

47:

190:

41:

4097:

6.17:

4.62:

89.21

1225039504:

90:

219:

28:

289:

2.97:

0.00:

97.03

1225039504:

482:

378:

158:

23:

579:

0.00:

13.86:

86.14

nicrec

nicrec is a utility part of K9toolkit, author Brendan Gregg, printing network traffic, Kb/s read and written. The recorder outputs its data under RRD format. nicrec used to observe the Kb/s transferred for all the network cards, including packet counts and average sizes. Recording points:

timestamp: seconds since Epoch
int: interface
rKB/s: read KB/s
wKB/s: write KB/s
rPk/s: read Packets/s
wPk/s: write Packets/s
rAvs: read Average size, bytes
wAvs: write Average size, bytes
%Util: utilisation (read+write)/nic speed
Sat: Saturation: defer, nocanput, norecvbuf, noxmtbuf

The output from nicrec is displayed below:

timestamp	interface	read KB/s	write KB/s	rPackets/s	wPackets/s	read average	write average	Util %	Sat %
1225354256:	e1000g0:	72.29:	3.07:	72.09:	41.56:	1026.89:	75.58:	0.06:	0.00:
1225354256:	mac:	72.29:	3.07:	72.09:	41.56:	1026.89:	75.58:	0.06:	0.00:

netrec

netrec is a utility, reporting TCP, UDP and IP statistics from a running local or global zone. If the system deploys one or more zones and if all zones share same TCP/IP stack then you can simple use -s flag to report the numbers just once. The recorder outputs its data under RRD format. netrec used to observe TCP, UDP and IP counters. Recording points:

timestamp: seconds since Epoch
udpInDatagrams
udpInErrors
udpOutDatagrams
udpOutErrors
tcpActiveOpens
tcpPassiveOpens
tcpAttemptFails
tcpEstabResets
tcpCurrEstab
tcpOutSegs
tcpOutDataSegs
tcpOutDataBytes
tcpRetransSegs
tcpRetransBytes
tcpListenDrop
tcpListenDropQ0
tcpHalfOpenDrop
tcpOutSackRetrans
ipInHdrErrors
ipInAddrErrors
ipInCksumErrs
tcpInErrs
udpInCksumErrs

The output from netrec is displayed below:

# netrec
global:1225354459:241756657:0:242661104:0:167385:475:54354:4477:17:20962546:11345630:4070136829:978714:913964897:0:0:0:435380:161:10
lobby:1225354459:241756657:0:242661104:0:167385:475:54354:4477:2:20962750:11345632:4070137181:978714:913964897:0:0:0:435380:161:10
wowza-test:1225354459:241756657:0:242661104:0:167385:475:54354:4477:0:20962750:11345634:4070137533:978714:913964897:0:0:0:435380:161:10
dss:1225354460:241756657:0:242661104:0:167385:475:54354:4477:0:20962750:11345636:4070137885:978714:913964897:0:0:0:435380:161:10
dss-test:1225354460:241756657:0:242661104:0:167385:475:54354:4477:0:20962750:11345638:4070138237:978714:913964897:0:0:0:435380:161:10

# netrec -s
global:1225354465:241756657:0:242661105:0:167386:475:54354:4477:16:20962798:11345713:4070158680:978715:913964897:0:0:0:435380:161:10
lobby:1225354465:2
wowza-test:1225354466:0
dss:1225354466:0
dss-test:1225354466:0

Note: If you observe for shared TCP/IP stacks we use -s flag in order to skip reporting same data for all zones

zonerec

zonerec is a simple script calling prstat utility to report zone utilisation in human readable format. This data needs to be parsed and prepared in RRD format. Future versions will include a new recorder which will output its data to RRD format. zonerec used to observe CPU and Mem utilisation, as reported by prstat The output from zonerec is displayed below:

# zonerec 60

2008-10-30:10:32:12 - 1225355532
ZONEID    NPROC  SWAP   RSS MEMORY      TIME  CPU ZONE                        
     0      110 1025M 1050M    26%  17:22:56  11% global                      
     3       28  140M  199M   4.9%   3:24:32 0.1% wowza-test                  
     1       32   47M   46M   1.1%   0:00:45 0.0% lobby                       
     4       23   38M   37M   0.9%   0:00:38 0.0% dss                         
Total: 224 processes, 1064 lwps, load averages: 0.30, 0.30, 0.29

Zone: global
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
  7323 sparvu    236M  157M cpu0    14    0   4:30:39 7.0% firefox-bin/12
   640 sparvu    217M  169M sleep   59    0   4:46:25 1.8% Xorg/1
   698 sparvu     64M   62M sleep   59    0   6:45:11 1.7% prstat/1
  7343 sparvu    445M  307M sleep   57    4   0:33:13 0.6% java/76
   689 sparvu   4008K 2024K sleep   49    0   0:12:42 0.1% cpubar.x86/1
  7280 sparvu    258M  159M sleep   57    4   0:03:21 0.0% java/36
 17860 root     6848K 3704K cpu1    50    0   0:00:00 0.0% prstat/1
 17694 sparvu   5596K 1940K sleep   59    0   0:00:00 0.0% sshd/1
   605 root     3532K 1280K sleep   59    0   0:00:00 0.0% sshd/1
 17696 sparvu   1208K 1012K sleep   59    0   0:00:00 0.0% ksh/1
   139 root     2248K 1332K sleep   59    0   0:00:00 0.0% syseventd/14
   149 root     6108K 3500K sleep   59    0   0:00:00 0.0% devfsadm/9
 NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU                             
    56 sparvu    936M  969M    24%  17:19:14  11%
    41 root      299M  359M   8.8%   0:02:42 0.1%
     4 postgres   15M   16M   0.4%   0:00:59 0.0%
     1 lp       1016K 3012K   0.1%   0:00:00 0.0%
     1 smmsp    3712K 6040K   0.1%   0:00:00 0.0%
     6 daemon     38M   29M   0.7%   0:00:01 0.0%
Total: 109 processes, 390 lwps, load averages: 0.30, 0.30, 0.29


Zone: lobby
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
 17654 sparvu   5716K 1956K sleep   59    0   0:00:00 0.0% sshd/1
 17686 sparvu   4852K 3100K sleep   59    0   0:00:00 0.0% ssh/1
  1123 root     3532K 1340K sleep   59    0   0:00:00 0.0% sshd/1
  1120 root     2088K 1204K sleep   59    0   0:00:00 0.0% ttymon/1
  1164 root     9180K 5760K sleep   59    0   0:00:03 0.0% snmpd/1
  1149 root     1740K  660K sleep   59    0   0:00:00 0.0% smcboot/1
  1039 root     4064K 3016K sleep   59    0   0:00:04 0.0% inetd/4
  1047 root     1108K  640K sleep   59    0   0:00:00 0.0% utmpd/1
   774 root        0K    0K sleep   60    -   0:00:00 0.0% zsched/1
   986 daemon   6048K 3484K sleep   59    0   0:00:01 0.0% nfsmapid/3
   981 daemon   2440K 1064K sleep   59    0   0:00:00 0.0% rpcbind/1
   796 root       11M 9648K sleep   59    0   0:00:07 0.0% svc.startd/14
 NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU                             
     3 sparvu    936M  969M    24%   0:00:00 0.0%
     1 smmsp    3712K 6040K   0.1%   0:00:00 0.0%
     6 daemon     38M   29M   0.7%   0:00:01 0.0%
    22 root      299M  359M   8.8%   0:00:44 0.0%
Total: 32 processes, 119 lwps, load averages: 0.30, 0.30, 0.29


Zone: wowza-test
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
  1379 root      167M  114M sleep   49    0   3:23:47 0.1% java/225
  1350 root     1128K  820K sleep   59    0   0:00:00 0.0% sh/1
  1743 daemon   2104K 1328K sleep   59    0   0:00:00 0.0% lockd/2
  1673 root     2128K 1168K sleep   59    0   0:00:00 0.0% ttymon/1
  1872 root     1740K  672K sleep   59    0   0:00:00 0.0% smcboot/1
  1870 root     1740K  932K sleep   59    0   0:00:00 0.0% smcboot/1
  1388 root     5016K 2668K sleep   59    0   0:00:11 0.0% nscd/28
  1382 daemon   3912K 2028K sleep   59    0   0:00:00 0.0% kcfd/3
  1256 root       10M 8956K sleep   59    0   0:00:11 0.0% svc.configd/20
  1249 root     2160K 1196K sleep   59    0   0:00:00 0.0% init/1
   947 root        0K    0K sleep   60    -   0:00:00 0.0% zsched/1
  1254 root       10M 9100K sleep   59    0   0:00:07 0.0% svc.startd/12
  1378 root     1100K  784K sleep   49    0   0:00:00 0.0% startup.sh/1
 NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU                             
    22 root      299M  359M   8.8%   3:24:32 0.1%
     6 daemon     38M   29M   0.7%   0:00:00 0.0%
Total: 28 processes, 330 lwps, load averages: 0.30, 0.30, 0.29



Zone: dss
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
  2050 root     2088K 1208K sleep   59    0   0:00:00 0.0% ttymon/1
  1752 daemon   3912K 2020K sleep   59    0   0:00:00 0.0% kcfd/3
  2000 daemon   2104K 1328K sleep   59    0   0:00:00 0.0% lockd/2
  2090 root     1740K  936K sleep   59    0   0:00:00 0.0% smcboot/1
  1750 root     5544K 2640K sleep   59    0   0:00:12 0.0% nscd/27
  1975 daemon   2440K 1012K sleep   59    0   0:00:00 0.0% rpcbind/1
  2018 root     2104K 1160K sleep   59    0   0:00:00 0.0% ttymon/1
  1983 daemon   2456K 1596K sleep   59    0   0:00:00 0.0% statd/1
  2042 root     1108K  644K sleep   59    0   0:00:00 0.0% utmpd/1
  2091 root     1740K  672K sleep   59    0   0:00:00 0.0% smcboot/1
  2028 root     3536K 1592K sleep   59    0   0:00:00 0.0% syslogd/13
  1985 daemon   2124K 1320K sleep   59    0   0:00:00 0.0% nfs4cbd/2
  1294 root       11M 8756K sleep   59    0   0:00:07 0.0% svc.startd/13
  1972 root     2424K 1004K sleep   59    0   0:00:00 0.0% cron/1
 NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU                             
     6 daemon     38M   29M   0.7%   0:00:01 0.0%
    17 root      299M  359M   8.8%   0:00:37 0.0%
Total: 23 processes, 97 lwps, load averages: 0.30, 0.30, 0.29

corerec

corerec is a utility using corestat, from Cooltools. The output is human readable format, it will require parsing and proper formating for RRD. corerec used to observe core utilisation from a T1 or T2 processor. Since T1 and T2 have different registers for keeping track of the usage the corestat utility has to be different for each case: corestat.t1 used for T1 processors and corestat.t2 for T2. Recording points:

%Usr: percentage user time
%Sys: percentage sys time
%Usr+Sys: both Usr + Sys
Avg: core utilisation

The output from corerec is displayed below:

For a T2 processor:
Core,Int-pipe     %Usr     %Sys     %Usr+Sys 
    0,0           0.05      0.37      0.42 
    0,1           0.06      0.07      0.13 
    1,0           0.04      0.08      0.12 
    1,1           0.01      0.05      0.06 
    2,0           0.09      0.11      0.20 
    2,1           0.01      0.06      0.07 
    3,0           0.02      0.15      0.17 
    3,1           0.01      0.05      0.05 
    4,0           0.01      0.13      0.14 
    4,1           0.01      0.04      0.05 
    5,0           0.07      0.10      0.16 
    5,1           0.01      0.45      0.46 
    6,0           0.02      0.10      0.12 
    6,1           0.01      0.06      0.07 
    7,0           0.03      0.12      0.14 
    7,1           0.01      0.05      0.06 
-------------     -----     -----    ------ 
    Avg           0.03      0.12      0.15

Important to note here is that utilisation for a T1 or T2 processor does not simple mean data from vmstat or mpstat alone. You have to use corerec in order to gather the correct utilisation figures. See below Ravindra Talashikar's notes about mpstat vmstat on T1 processors!

jvmrec

jvmrec is a utility based on jstat, part of JDK helping to extract Garbage Collection statistics from a running virtual machine. This recorder will loop over all running zones found on the system and will fetch each java process found and extract its GC numbers. The recorder outputs its data under RRD format. jvmrec records the GC statistics useful to understand how your JVMs are running. Recording points:

zone.pid: name of the zone and process ID
timestamp: seconds since Epoch
S0%: Survivor s0 utilisation, percentage of the space's current capacity
S1%: Survivor s1 utilisation, percentage of the space's current capacity
Eden%: Eden space utilisation, percentage of the space's current capacity
Old%: Old space utilisation, percentage of the space's current capacity
Perm%: Permanent space utilisation, percentage of the space's current capacity
No.mGC: Number of young generation GC events
Time.mGC: Young generation garbage collection time, secs
No.MGC: Number of full GC events
Time.MGC: Full garbage collection time, secs
TotalGC: Total garbage collection time, secs

The output from jvmrec is displayed below:

zone.pid	timestamp	S0%	S1%	Eden%	Old%	Perm%	No.mGC	Time.mGC	No.MGC	Time.MGC	Total GC
global.23699:	1225360607:	0.00:	29.51:	10.03:	10.07:	60.44:	9:	0.100:	1:	0.058:	0.158
global.23699:	1225360668:	0.00:	29.51:	12.76:	10.07:	60.44:	9:	0.100:	1:	0.058:	0.158

webrec

webrec is a utility based on Apache HTTPClient, written in Java intended to be used to measure response times for different HTTP actions: POST, GET. Currently webrec supports only GET HTTP methods. Work in under way to expand this to POST methods and authentication.

webrec checks for a master configuration file, called: webrec.conf located under SDR/etc/ director, where SDR Recording module has been installed. It parses the configuration file: webrec.conf and it retrieves the values for timeout and the name definitions of each workload. For each workload it creates a thread and its starting to follow each URL recording the response times. WebRec supports Cookies as defined in RFC 2965

webrec operates against one or many workloads. Workload defines a sequence of URLs with the following properties:

name : name of the workload
delay : delay after each URL (in seconds)
interval: the interval to invoke workloads (in seconds)
timeout : determines the timeout until a connection is etablished. A value of zero means the timeout is not used. The default value is zero. (in milliseconds)

The output from webrec is displayed below:

timestamp	transaction name	response time (ms)
1273024907:	main:	0.352
1273024912:	recdesign:	0.355
1273024917:	recorders:	0.381
1273024922:	reporting:	0.318
1273024927:	cpuplayer:	0.399
1273024932:	relnotes:	1.174
1273024937:	docs:	0.298
1273024942:	faq:	0.366

The webrec.conf contains a single workload, called sdr with 8 transactions, each performed after Z=5 seconds. Delay parameter defines the number of seconds between each transaction.

<workload name="sdr" delay="5" interval="60" timeout="120">

    <transaction
        id="main"
        method="GET"
        url="http://www.systemdatarecorder.org/"
        value="xxx"/>

    <:transaction
        id="recdesign"
        method="GET"
        url="http://www.systemdatarecorder.org/recording/recdesign.html"
        value="xxx"/>

    <transaction
        id="recorders"
        method="GET"
        url="http://www.systemdatarecorder.org/recording/recorders.html"
        value="xxx"/>

    <transaction
        id="reporting"
        method="GET"
        url="http://www.systemdatarecorder.org/reporting/"
        value="xxx"/>

    <transaction
        id="cpuplayer"
        method="GET"
        url="http://www.systemdatarecorder.org/cpuplayer/"
        value="xxx"/>

    <transaction
        id="relnotes"
        method="GET"
        url="http://www.systemdatarecorder.org/man/relnotes.html"
        value="xxx"/>

    <transaction
        id="docs"
        method="GET"
        url="http://www.systemdatarecorder.org/man/index.html"
        value="xxx"/>>

    <transaction
        id="faq"
        method="GET"
        url="http://www.systemdatarecorder.org/man/faq.html"
        value="xxx"/>

</workload>

At this moment webrec.conf must be manually written. Plans are to reuse the test plan generated by Apache jmeter to webrec, to easy the task of recording a series of transactions. You can however even now, record your workloads using jmeter and convert that to webrec or manually define the workload definition.

Last updated: 2010-05-16

Back to main homepage