Recorders
The recording module consists of the following data collectors:| Item | Description | Based On | FreeBSD | RedHat | (Open)Solaris |
| sysrec | system utilisation and queuing | Perl5, Kstat | n/a | n/a | yes |
| cpurec | per-cpu detailed statistics | Perl5, Kstat | n/a | n/a | yes |
| nicrec | network interface statistics | Perl5, Kstat | n/a | yes | yes |
| netrec | network protocol statistics | Perl5, netstat | n/a | n/a | yes |
| zonerec | zone statistics | Ksh, prstat | n/a | n/a | yes |
| corerec | CMT T1, T2 processor statistics | Ksh, Perl5, cpustat | n/a | n/a | yes |
| jvmrec | garbage collection statistics | Ksh, jstat | yes | yes | yes |
| webrec | HTTP response time recorder | Java, Apache HTTPClient | yes | yes | yes |
sysrec
sysrec is a utility, part of K9Toolkit, author Brendan Gregg. The toolkit is a collection of free Perl scripts used to troubleshoot and observe (Open)Solaris systems. Check Appendix for more details. The recorder has been modified to output its data into RRD format. sysrec records system utilisation and certain additional kernel statistics used as a starting point in observing the system's general health. The output from sysrec is displayed below:| timestamp | CPU Util % | Mem Util % | Disk Util % | Net Util % | Run Queue | Scan rate | Disk waits | Net errors |
| 1225038537: | 10.56: | 70.79: | 9.87: | 0.13: | 0.02: | 0.02: | 0.19: | 0.00 |
| 1225038539: | 3.92: | 70.79: | 0.00: | 0.00: | 0.00: | 0.00: | 0.00: | 0.00 |
| 1225038538: | 2.94: | 70.79: | 0.00: | 0.00: | 0.00: | 0.00: | 0.00: | 0.00 |
cpurec
cpurec is a utility, collecting per-CPU data from kstat The recorder outputs its data under RRD format. cpurec used mainly to observe CPU activity and analyse how the CPUs are used in the system. Useful for capacity planning. Recording points:timestamp: seconds since Epoch
Xcalls: rate of multiprocessor cross calls
Intr: rate of intrerrupts
iThr: rate of interrupts threads
Csw: rate of context switches
Icsw: rate of involuntary context switches
Migr: rate of migrations
Smtx: rate of kernel mutexes
Syscalls: rate of system calls
User: percentage of time spent in user mode
Sys: percentage of time spent in sys mode
Idle: percentage of time spent in idle mode
| timestamp | Cpuid | Xcalls | Intr | iThr | Csw | Icsw | Migr | Smtx | Syscalls | User % | Sys % | Idle % |
| 1225039500: | 1: | 97: | 568: | 49: | 1195: | 69: | 190: | 40: | 4183: | 5.95: | 4.37: | 89.68 |
| 1225039500: | 0: | 98: | 936: | 513: | 1169: | 47: | 190: | 41: | 4097: | 6.17: | 4.62: | 89.21 |
| 1225039504: | 1: | 0: | 90: | 3: | 219: | 1: | 28: | 2: | 289: | 2.97: | 0.00: | 97.03 |
| 1225039504: | 0: | 0: | 482: | 378: | 158: | 3: | 23: | 3: | 579: | 0.00: | 13.86: | 86.14 |
nicrec
nicrec is a utility part of K9toolkit, author Brendan Gregg, printing network traffic, Kb/s read and written. The recorder outputs its data under RRD format. nicrec used to observe the Kb/s transferred for all the network cards, including packet counts and average sizes. Recording points:timestamp: seconds since Epoch
int: interface
rKB/s: read KB/s
wKB/s: write KB/s
rPk/s: read Packets/s
wPk/s: write Packets/s
rAvs: read Average size, bytes
wAvs: write Average size, bytes
%Util: utilisation (read+write)/nic speed
Sat: Saturation: defer, nocanput, norecvbuf, noxmtbuf
| timestamp | interface | read KB/s | write KB/s | rPackets/s | wPackets/s | read average | write average | Util % | Sat % |
| 1225354256: | e1000g0: | 72.29: | 3.07: | 72.09: | 41.56: | 1026.89: | 75.58: | 0.06: | 0.00: |
| 1225354256: | mac: | 72.29: | 3.07: | 72.09: | 41.56: | 1026.89: | 75.58: | 0.06: | 0.00: |
netrec
netrec is a utility, reporting TCP, UDP and IP statistics from a running local or global zone. If the system deploys one or more zones and if all zones share same TCP/IP stack then you can simple use -s flag to report the numbers just once. The recorder outputs its data under RRD format. netrec used to observe TCP, UDP and IP counters. Recording points:timestamp: seconds since Epoch
udpInDatagrams
udpInErrors
udpOutDatagrams
udpOutErrors
tcpActiveOpens
tcpPassiveOpens
tcpAttemptFails
tcpEstabResets
tcpCurrEstab
tcpOutSegs
tcpOutDataSegs
tcpOutDataBytes
tcpRetransSegs
tcpRetransBytes
tcpListenDrop
tcpListenDropQ0
tcpHalfOpenDrop
tcpOutSackRetrans
ipInHdrErrors
ipInAddrErrors
ipInCksumErrs
tcpInErrs
udpInCksumErrs
# netrec global:1225354459:241756657:0:242661104:0:167385:475:54354:4477:17:20962546:11345630:4070136829:978714:913964897:0:0:0:435380:161:10 lobby:1225354459:241756657:0:242661104:0:167385:475:54354:4477:2:20962750:11345632:4070137181:978714:913964897:0:0:0:435380:161:10 wowza-test:1225354459:241756657:0:242661104:0:167385:475:54354:4477:0:20962750:11345634:4070137533:978714:913964897:0:0:0:435380:161:10 dss:1225354460:241756657:0:242661104:0:167385:475:54354:4477:0:20962750:11345636:4070137885:978714:913964897:0:0:0:435380:161:10 dss-test:1225354460:241756657:0:242661104:0:167385:475:54354:4477:0:20962750:11345638:4070138237:978714:913964897:0:0:0:435380:161:10 # netrec -s global:1225354465:241756657:0:242661105:0:167386:475:54354:4477:16:20962798:11345713:4070158680:978715:913964897:0:0:0:435380:161:10 lobby:1225354465:2 wowza-test:1225354466:0 dss:1225354466:0 dss-test:1225354466:0 |
zonerec
zonerec is a simple script calling prstat utility to report zone utilisation in human readable format. This data needs to be parsed and prepared in RRD format. Future versions will include a new recorder which will output its data to RRD format. zonerec used to observe CPU and Mem utilisation, as reported by prstat The output from zonerec is displayed below:
# zonerec 60
2008-10-30:10:32:12 - 1225355532
ZONEID NPROC SWAP RSS MEMORY TIME CPU ZONE
0 110 1025M 1050M 26% 17:22:56 11% global
3 28 140M 199M 4.9% 3:24:32 0.1% wowza-test
1 32 47M 46M 1.1% 0:00:45 0.0% lobby
4 23 38M 37M 0.9% 0:00:38 0.0% dss
Total: 224 processes, 1064 lwps, load averages: 0.30, 0.30, 0.29
Zone: global
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
7323 sparvu 236M 157M cpu0 14 0 4:30:39 7.0% firefox-bin/12
640 sparvu 217M 169M sleep 59 0 4:46:25 1.8% Xorg/1
698 sparvu 64M 62M sleep 59 0 6:45:11 1.7% prstat/1
7343 sparvu 445M 307M sleep 57 4 0:33:13 0.6% java/76
689 sparvu 4008K 2024K sleep 49 0 0:12:42 0.1% cpubar.x86/1
7280 sparvu 258M 159M sleep 57 4 0:03:21 0.0% java/36
17860 root 6848K 3704K cpu1 50 0 0:00:00 0.0% prstat/1
17694 sparvu 5596K 1940K sleep 59 0 0:00:00 0.0% sshd/1
605 root 3532K 1280K sleep 59 0 0:00:00 0.0% sshd/1
17696 sparvu 1208K 1012K sleep 59 0 0:00:00 0.0% ksh/1
139 root 2248K 1332K sleep 59 0 0:00:00 0.0% syseventd/14
149 root 6108K 3500K sleep 59 0 0:00:00 0.0% devfsadm/9
NPROC USERNAME SWAP RSS MEMORY TIME CPU
56 sparvu 936M 969M 24% 17:19:14 11%
41 root 299M 359M 8.8% 0:02:42 0.1%
4 postgres 15M 16M 0.4% 0:00:59 0.0%
1 lp 1016K 3012K 0.1% 0:00:00 0.0%
1 smmsp 3712K 6040K 0.1% 0:00:00 0.0%
6 daemon 38M 29M 0.7% 0:00:01 0.0%
Total: 109 processes, 390 lwps, load averages: 0.30, 0.30, 0.29
Zone: lobby
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
17654 sparvu 5716K 1956K sleep 59 0 0:00:00 0.0% sshd/1
17686 sparvu 4852K 3100K sleep 59 0 0:00:00 0.0% ssh/1
1123 root 3532K 1340K sleep 59 0 0:00:00 0.0% sshd/1
1120 root 2088K 1204K sleep 59 0 0:00:00 0.0% ttymon/1
1164 root 9180K 5760K sleep 59 0 0:00:03 0.0% snmpd/1
1149 root 1740K 660K sleep 59 0 0:00:00 0.0% smcboot/1
1039 root 4064K 3016K sleep 59 0 0:00:04 0.0% inetd/4
1047 root 1108K 640K sleep 59 0 0:00:00 0.0% utmpd/1
774 root 0K 0K sleep 60 - 0:00:00 0.0% zsched/1
986 daemon 6048K 3484K sleep 59 0 0:00:01 0.0% nfsmapid/3
981 daemon 2440K 1064K sleep 59 0 0:00:00 0.0% rpcbind/1
796 root 11M 9648K sleep 59 0 0:00:07 0.0% svc.startd/14
NPROC USERNAME SWAP RSS MEMORY TIME CPU
3 sparvu 936M 969M 24% 0:00:00 0.0%
1 smmsp 3712K 6040K 0.1% 0:00:00 0.0%
6 daemon 38M 29M 0.7% 0:00:01 0.0%
22 root 299M 359M 8.8% 0:00:44 0.0%
Total: 32 processes, 119 lwps, load averages: 0.30, 0.30, 0.29
Zone: wowza-test
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
1379 root 167M 114M sleep 49 0 3:23:47 0.1% java/225
1350 root 1128K 820K sleep 59 0 0:00:00 0.0% sh/1
1743 daemon 2104K 1328K sleep 59 0 0:00:00 0.0% lockd/2
1673 root 2128K 1168K sleep 59 0 0:00:00 0.0% ttymon/1
1872 root 1740K 672K sleep 59 0 0:00:00 0.0% smcboot/1
1870 root 1740K 932K sleep 59 0 0:00:00 0.0% smcboot/1
1388 root 5016K 2668K sleep 59 0 0:00:11 0.0% nscd/28
1382 daemon 3912K 2028K sleep 59 0 0:00:00 0.0% kcfd/3
1256 root 10M 8956K sleep 59 0 0:00:11 0.0% svc.configd/20
1249 root 2160K 1196K sleep 59 0 0:00:00 0.0% init/1
947 root 0K 0K sleep 60 - 0:00:00 0.0% zsched/1
1254 root 10M 9100K sleep 59 0 0:00:07 0.0% svc.startd/12
1378 root 1100K 784K sleep 49 0 0:00:00 0.0% startup.sh/1
NPROC USERNAME SWAP RSS MEMORY TIME CPU
22 root 299M 359M 8.8% 3:24:32 0.1%
6 daemon 38M 29M 0.7% 0:00:00 0.0%
Total: 28 processes, 330 lwps, load averages: 0.30, 0.30, 0.29
Zone: dss
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
2050 root 2088K 1208K sleep 59 0 0:00:00 0.0% ttymon/1
1752 daemon 3912K 2020K sleep 59 0 0:00:00 0.0% kcfd/3
2000 daemon 2104K 1328K sleep 59 0 0:00:00 0.0% lockd/2
2090 root 1740K 936K sleep 59 0 0:00:00 0.0% smcboot/1
1750 root 5544K 2640K sleep 59 0 0:00:12 0.0% nscd/27
1975 daemon 2440K 1012K sleep 59 0 0:00:00 0.0% rpcbind/1
2018 root 2104K 1160K sleep 59 0 0:00:00 0.0% ttymon/1
1983 daemon 2456K 1596K sleep 59 0 0:00:00 0.0% statd/1
2042 root 1108K 644K sleep 59 0 0:00:00 0.0% utmpd/1
2091 root 1740K 672K sleep 59 0 0:00:00 0.0% smcboot/1
2028 root 3536K 1592K sleep 59 0 0:00:00 0.0% syslogd/13
1985 daemon 2124K 1320K sleep 59 0 0:00:00 0.0% nfs4cbd/2
1294 root 11M 8756K sleep 59 0 0:00:07 0.0% svc.startd/13
1972 root 2424K 1004K sleep 59 0 0:00:00 0.0% cron/1
NPROC USERNAME SWAP RSS MEMORY TIME CPU
6 daemon 38M 29M 0.7% 0:00:01 0.0%
17 root 299M 359M 8.8% 0:00:37 0.0%
Total: 23 processes, 97 lwps, load averages: 0.30, 0.30, 0.29
|
corerec
corerec is a utility using corestat, from Cooltools. The output is human readable format, it will require parsing and proper formating for RRD. corerec used to observe core utilisation from a T1 or T2 processor. Since T1 and T2 have different registers for keeping track of the usage the corestat utility has to be different for each case: corestat.t1 used for T1 processors and corestat.t2 for T2. Recording points:%Usr: percentage user time
%Sys: percentage sys time
%Usr+Sys: both Usr + Sys
Avg: core utilisation
For a T2 processor:
Core,Int-pipe %Usr %Sys %Usr+Sys
0,0 0.05 0.37 0.42
0,1 0.06 0.07 0.13
1,0 0.04 0.08 0.12
1,1 0.01 0.05 0.06
2,0 0.09 0.11 0.20
2,1 0.01 0.06 0.07
3,0 0.02 0.15 0.17
3,1 0.01 0.05 0.05
4,0 0.01 0.13 0.14
4,1 0.01 0.04 0.05
5,0 0.07 0.10 0.16
5,1 0.01 0.45 0.46
6,0 0.02 0.10 0.12
6,1 0.01 0.06 0.07
7,0 0.03 0.12 0.14
7,1 0.01 0.05 0.06
------------- ----- ----- ------
Avg 0.03 0.12 0.15
|
jvmrec
jvmrec is a utility based on jstat, part of JDK helping to extract Garbage Collection statistics from a running virtual machine. This recorder will loop over all running zones found on the system and will fetch each java process found and extract its GC numbers. The recorder outputs its data under RRD format. jvmrec records the GC statistics useful to understand how your JVMs are running. Recording points:zone.pid: name of the zone and process ID
timestamp: seconds since Epoch
S0%: Survivor s0 utilisation, percentage of the space's current capacity
S1%: Survivor s1 utilisation, percentage of the space's current capacity
Eden%: Eden space utilisation, percentage of the space's current capacity
Old%: Old space utilisation, percentage of the space's current capacity
Perm%: Permanent space utilisation, percentage of the space's current capacity
No.mGC: Number of young generation GC events
Time.mGC: Young generation garbage collection time, secs
No.MGC: Number of full GC events
Time.MGC: Full garbage collection time, secs
TotalGC: Total garbage collection time, secs
| zone.pid | timestamp | S0% | S1% | Eden% | Old% | Perm% | No.mGC | Time.mGC | No.MGC | Time.MGC | Total GC |
| global.23699: | 1225360607: | 0.00: | 29.51: | 10.03: | 10.07: | 60.44: | 9: | 0.100: | 1: | 0.058: | 0.158 |
| global.23699: | 1225360668: | 0.00: | 29.51: | 12.76: | 10.07: | 60.44: | 9: | 0.100: | 1: | 0.058: | 0.158 |
webrec
webrec is a utility based on Apache HTTPClient, written in Java intended to be used to measure response times for different HTTP actions: POST, GET. Currently webrec supports only GET HTTP methods. Work in under way to expand this to POST methods and authentication.webrec checks for a master configuration file, called: webrec.conf located under SDR/etc/ director, where SDR Recording module has been installed. It parses the configuration file: webrec.conf and it retrieves the values for timeout and the name definitions of each workload. For each workload it creates a thread and its starting to follow each URL recording the response times. WebRec supports Cookies as defined in RFC 2965
webrec operates against one or many workloads. Workload defines a sequence of URLs with the following properties:
- name : name of the workload
- delay : delay after each URL (in seconds)
- interval: the interval to invoke workloads (in seconds)
- timeout : determines the timeout until a connection is etablished. A value of zero means the timeout is not used. The default value is zero. (in milliseconds)
The output from webrec is displayed below:
| timestamp | transaction name | response time (ms) |
| 1273024907: | main: | 0.352 |
| 1273024912: | recdesign: | 0.355 |
| 1273024917: | recorders: | 0.381 |
| 1273024922: | reporting: | 0.318 |
| 1273024927: | cpuplayer: | 0.399 |
| 1273024932: | relnotes: | 1.174 |
| 1273024937: | docs: | 0.298 |
| 1273024942: | faq: | 0.366 |
The webrec.conf contains a single workload, called sdr with 8 transactions, each performed after Z=5 seconds. Delay parameter defines the number of seconds between each transaction.
<workload name="sdr" delay="5" interval="60" timeout="120">
<transaction
id="main"
method="GET"
url="http://www.systemdatarecorder.org/"
value="xxx"/>
<:transaction
id="recdesign"
method="GET"
url="http://www.systemdatarecorder.org/recording/recdesign.html"
value="xxx"/>
<transaction
id="recorders"
method="GET"
url="http://www.systemdatarecorder.org/recording/recorders.html"
value="xxx"/>
<transaction
id="reporting"
method="GET"
url="http://www.systemdatarecorder.org/reporting/"
value="xxx"/>
<transaction
id="cpuplayer"
method="GET"
url="http://www.systemdatarecorder.org/cpuplayer/"
value="xxx"/>
<transaction
id="relnotes"
method="GET"
url="http://www.systemdatarecorder.org/man/relnotes.html"
value="xxx"/>
<transaction
id="docs"
method="GET"
url="http://www.systemdatarecorder.org/man/index.html"
value="xxx"/>>
<transaction
id="faq"
method="GET"
url="http://www.systemdatarecorder.org/man/faq.html"
value="xxx"/>
</workload>
|
At this moment webrec.conf must be manually written. Plans are to reuse the test plan generated by Apache jmeter to webrec, to easy the task of recording a series of transactions. You can however even now, record your workloads using jmeter and convert that to webrec or manually define the workload definition.
Last updated: 2010-05-16
Back to main homepage