Recorders

The recording module consists of the following data collectors:

Item Description Based On FreeBSD Linux Solaris
sysrec overall system statistics Perl5 n/a yes yes
cpurec per-cpu statistics Perl5 n/a yes yes
diskrec per-disk statistics Perl5 n/a n/a n/a
nicrec per-NIC statistics Perl5 n/a yes yes
corerec CMT T1, T2 processor statistics Ksh, Perl5, cpustat n/a n/a yes
jvmrec garbage collection statistics Perl5/Ksh, jstat yes yes yes
netrec network protocol statistics Perl5, netstat n/a yes yes
procrec per process statistics Perl5 n/a yes n/a
webrec HTTP response time recorder Java, Apache HTTPClient yes yes yes
zonerec zone statistics Ksh, prstat n/a n/a yes

sysrec

Linux

Data Source /proc
API Sys::Statistics::Linux, Perl5

sysrec records system metrics from a Linux x86/64 kernel system. The recorder requires a minimum version of Linux kernel: 2.6 ! The recorder uses Sys::Statistics::Linux a perl module which handles data collection from /proc interface. The recorder has been modified to output its data as RRD.

sysrec records system utilisation along with certain other imortant metrics:

timestamp cpu% mem% swap% readReq writeReq totReq readByt writeByt totByt rxByt txByt ntByt
1293282411: 0.00: 98.15: 0.00: 0: 1: 1: 0.00: 102.97: 102.97: 0.19: 0.48: 0.66:

user% nice% sys% idle% iowait% irq% softirq% steal% rxerr txerr rxdrp txdrp avg1 avg5 avg15
0.00: 0.00: 0.00: 100.00: 0.00: 0.00: 0.00: 0.00: 0: 0: 0: 0: 0.11: 0.03: 0.01

  • timestamp: seconds since Epoch
  • cpu total: total percentage of CPU utilization
  • memusedper: total size of used memory in percent
  • swapusedper: total size of used swap space in percent
  • readReq: total disk read requests, counter
  • writeReq: total disk write requests, counter
  • totReq: total disk read+write requests, counter
  • readByt: total read bytes / sec, in KB
  • writeByt: total write bytes / sec, in KB
  • totByt: total read+write bytes / sec, in KB
  • rxByt: total received bytes /sec, in KB
  • txByt: total transmitted bytes /sec, in KB
  • ntByt: total received + transmitted bytes /sec, in KB
  • user: CPU Utilisation User space in percent
  • nice: CPU utilisation User space with nice priority
  • system: CPU utilisation System space
  • idle: CPU Utilisation idle state
  • iowait: CPU Percentage in idle state because an I/O operation is waiting to complete
  • irq: CPU Percentage servicing interrupts
  • softirq: CPU Percentage servicing softirqs
  • steal: CPU Percentage of time spent in other operating systems when running in a virtualized environment
  • rxerr: Number of errors that happend while received packets/second
  • txerr: Number of errors that happend while transmitting packets/second
  • rxdrp: Number of rx packets that were dropped per second
  • txdrp: Number of rx packets that were dropped per second
  • avg_1: LA of the last minute
  • avg_5: LA of the last 5 minutes
  • avg_15: LA of the last 15 minutes

Solaris

Data Source kstat
API Sun::Solaris::Kstat, Perl5

sysrec is a utility, part of K9Toolkit, author Brendan Gregg. The toolkit is a collection of free Perl scripts used to troubleshoot and observe (Open)Solaris systems. Check Appendix for more details. The recorder has been modified to output its data as RRD.

sysrec records system utilisation and certain additional kernel statistics used as a starting point in observing the system's general health. The output from sysrec is displayed below:

timestamp CPU Util% Mem Util% Disk Util% Net Util% Run Queue Scan rate Disk waits Net errors Usr Util% Sys Util% Idle Util% Avg_1 Avg_5 Avg_15
1294052332: 12.02: 69.54: 0.00: 0.00: 0.00: 0.00: 0.00: 0.00: 0.00: 12.02: 87.98: 0.00: 0.00: 0.00
1294052764: 99.26: 72.83: 0.00: 0.00: 0.00: 0.00: 0.00: 0.00: 95.25: 4.01: 0.74: 0.25: 0.06: 0.02

Sometimes you might need to ask sysrec to monitor your systems using a sub-second interval. What is this ? Interval is usually the input option to control the interval in seconds between each sample. SDR versions before 0.71 did not support interval lower than 1 sec. In latest releases we enhanced the sleeping between samples, using signals in Perl and we obtain a much finer degree of sleeping between each sample. See below: 

Interval 1sec, 5 times:
1313952055:9.09:98.18:0.00:0:0:0:0.00:0.00:0.00:0.00:0.00:0.00:9.09:0.00:0.00:90.91:0.00:0.00:0.00:0.00:0:0:0:0:0.00:0.00:0.00
1313952056:1.00:98.18:0.00:0:0:0:0.00:0.00:0.00:0.06:0.24:0.30:0.00:0.00:1.00:99.00:0.00:0.00:0.00:0.00:0:0:0:0:0.00:0.00:0.00
1313952057:0.00:98.20:0.00:0:0:0:0.00:0.00:0.00:0.06:0.24:0.30:0.00:0.00:0.00:100.00:0.00:0.00:0.00:0.00:0:0:0:0:0.00:0.00:0.00
1313952058:0.99:98.20:0.00:0:0:0:0.00:0.00:0.00:0.06:0.24:0.30:0.99:0.00:0.00:99.01:0.00:0.00:0.00:0.00:0:0:0:0:0.00:0.00:0.00
1313952059:0.00:98.20:0.00:0:0:0:0.00:0.00:0.00:0.06:0.24:0.30:0.00:0.00:0.00:100.00:0.00:0.00:0.00:0.00:0:0:0:0:0.00:0.00:0.00

Interval 0.5sec, 5 times:
1313952100.208:0.00:98.18:0.00:0:0:0:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0.00:100.00:0.00:0.00:0.00:0.00:0:0:0:0:0.00:0.00:0.00
1313952100.710:1.96:98.18:0.00:0:0:0:0.00:0.00:0.00:0.13:0.47:0.60:1.96:0.00:0.00:98.04:0.00:0.00:0.00:0.00:0:0:0:0:0.00:0.00:0.00
1313952101.210:0.00:98.20:0.00:0:0:0:0.00:0.00:0.00:0.13:0.47:0.60:0.00:0.00:0.00:100.00:0.00:0.00:0.00:0.00:0:0:0:0:0.00:0.00:0.00
1313952101.710:1.96:98.20:0.00:0:0:0:0.00:0.00:0.00:0.13:0.47:0.60:1.96:0.00:0.00:98.04:0.00:0.00:0.00:0.00:0:0:0:0:0.00:0.00:0.00
1313952102.210:0.00:98.20:0.00:0:0:0:0.00:0.00:0.00:0.13:0.47:0.60:0.00:0.00:0.00:100.00:0.00:0.00:0.00:0.00:0:0:0:0:0.00:0.00:0.00

See the timestamp reporting a different time resolution. Be careful: dont play in PROD systems with values very low since the recorders will add an overhead to your system. Remember SDR is not DTrace or SystemTap !

cpurec

Linux

Data Source /proc
API Sys::Statistics::Linux, Perl5

cpurec records per CPU metrics from a Linux x86/64 system. The recorder requires a minimum version of Linux kernel: 2.6 ! The recorder uses Sys::Statistics::Linux a perl module which handles data collection from /proc interface. The recorder has been modified to output its data as RRD.

cpurec records system utilisation along with certain other imortant metrics:

timestamp cpuid user nice system idle total irq softirq iowait steal
1293371032: cpu0: 0.00: 0.00: 0.00: 85.15: 14.85: 0.00: 14.85: 0.00: 0.00:
1293371032: cpu1: 0.00: 0.00: 0.00: 91.09: 8.91: 0.00: 8.91: 0.00: 0.00:

  • timestamp:seconds since Epoch
  • cpuid: CPU id
  • user: CPU Utilisation User space percentage
  • nice: CPU utilisation User space with nice priority percentage
  • system: CPU utilisation System space percentage
  • idle: CPU Utilisation idle state percentage
  • total: Total CPU Utilisation percentage
  • irq: CPU Percentage servicing interrupts
  • softirq: CPU Percentage servicing softirqs
  • iowait: CPU Percentage in idle state because an I/O operation is waiting to complete
  • steal: CPU Percentage of time spent in other operating systems when running in a virtualized environment

Solaris

Data Source kstat
API Sun::Solaris::Kstat, Perl5

cpurec is a utility, collecting per-CPU data from kstat. The recorder outputs its data under RRD format. cpurec used mainly to observe CPU activity and analyse how the CPUs are used in the system. Useful for capacity planning. Recording points:

  • timestamp: seconds since Epoch
  • Xcalls: rate of multiprocessor cross calls
  • Intr: rate of intrerrupts
  • iThr: rate of interrupts threads
  • Csw: rate of context switches
  • Icsw: rate of involuntary context switches
  • Migr: rate of migrations
  • Smtx: rate of kernel mutexes
  • Syscalls: rate of system calls
  • User: percentage of time spent in user mode
  • Sys: percentage of time spent in sys mode
  • Idle: percentage of time spent in idle mode

The output from cpurec is displayed below:

timestamp Cpuid Xcalls Intr iThr Csw Icsw Migr Smtx Syscalls User % Sys % Idle %
1225039500: 1: 97: 568: 49: 1195: 69: 190: 40: 4183: 5.95: 4.37: 89.68
1225039500: 0: 98: 936: 513: 1169: 47: 190: 41: 4097: 6.17: 4.62: 89.21
1225039504: 1: 0: 90: 3: 219: 1: 28: 2: 289: 2.97: 0.00: 97.03
1225039504: 0: 0: 482: 378: 158: 3: 23: 3: 579: 0.00: 13.86: 86.14

nicrec

Linux

Data Source /proc
API Sys::Statistics::Linux, Perl5

nicrec records per NIC statistics, receiving and transmitting metrics: the no. of KBytes received/sec, the no. of packets received/sec, the no. of KBytes transmitted/sec, the no. of packets transmitted/sec and many other metrics. The recorder uses Sys::Statistics::Linux to fetch all metrics. The recorder has been modified to output its data as RRD and no delta values are stored, but rather raw data.

nicrec records per NIC metrics:

  • timestamp: seconds since Epoch
  • interface: NIC, interface
  • rxKB: the no. of KBytes received/sec
  • rxpcks: the no. of packets received/sec
  • rxerrs: the no. of errors while received packets/sec
  • rxdrop: the no. of packets that were dropped/sec
  • rxfifo: the no. of FIFO overruns on received packets/sec
  • rxframe: the no. of carrier errors on received packets/sec
  • rxcompr: the no. of compressed packets received/sec
  • rxmulti: the no. of multicast packets received/sec
  • txKB: the no. of KBytes transmitted/sec
  • txpcks: the no. of packets transmitted/sec
  • txerrs: the no. of errors transmitting packets/sec
  • txdrop: the no. of packets that were dropped/sec
  • txfifo: the no. of FIFO overruns on transmitted packets/sec
  • txcolls: the no. of collisions that were detected/sec
  • txcarr: the no. of carrier errors on transmitted packets/sec
  • txcompr: the no. of compressed packets transmitted/sec
  • ttpcks: the no. of total packets (received + transmitted)/sec
  • ttKB: the no. of total KBytes (received + transmitted)/sec

The output from nicrec is displayed below:

timestamp interface rxKB rxpcks rxerrs rxdrop rxfifo rxframe rxcompr rxmulti
1313536832: wlan0: 0.06: 1: 0: 0: 0: 0: 0: 0:

txKB txpcks txerrs txdrop txfifo txcolls txcarr txcompr ttpcks ttKB
0.00: 0: 0: 0: 0: 0: 0: 0: 1: 0.06

Solaris

Data Source kstat
API Sun::Solaris::Kstat, Perl5

nicrec is a utility part of K9toolkit, author Brendan Gregg, printing network traffic, KB/s read and written. The recorder outputs its data under RRD format. A new version, a native C implementation, developed by Tim Cook of Sun Microsystems was offered some time ago. SDR deploys Tim's recorders since it is much lighter and better supported under Solaris and Linux systems. You can however have access to the original Perl version too.

nicrec used to observe the Kb/s transferred for all the network cards, including packet counts and average sizes. Recording points:

  • timestamp: seconds since Epoch
  • int: interface
  • rKB/s: read KB/s
  • wKB/s: write KB/s
  • rPk/s: read Packets/s
  • wPk/s: write Packets/s
  • rAvs: read Average size, bytes
  • wAvs: write Average size, bytes
  • %Util: utilisation (read+write)/nic speed
  • Sat: Saturation: defer, nocanput, norecvbuf, noxmtbuf

The output from nicrec is displayed below:

timestamp interface read KB/s write KB/s rPackets/s wPackets/s read average write average Util % Sat %
1225354256: e1000g0: 72.29: 3.07: 72.09: 41.56: 1026.89: 75.58: 0.06: 0.00:
1225354256: mac: 72.29: 3.07: 72.09: 41.56: 1026.89: 75.58: 0.06: 0.00:

netrec

Linux

Data Source netrec
API Perl5

netrec records per protocol statistics TCP, UDP from netstat. The recorder uses netstat binary, so this utulity should be present on your Linux installation. The recorder has been modified to output its data as RRD and no delta values are stored, but rather raw data.

netrec records TCP, UDP metrics:

timestamp active passive failed recv_rst est sent_rst upks_recv upks_sent
1293441496: 52: 20: 18: 0: 2: 25: 2009: 1616

  • timestamp:seconds since Epoch
  • active: TCP active connections
  • passive: TCP passive connections
  • failed: TCP failed connection attempts
  • recv_rst: TCP connection resets received
  • est: TCP connections established
  • sent_rst: TCP resets sent
  • upks_recv: UDP packets received
  • upks_sent: UDP packets sent

Solaris

Data Source netrec
API Perl5

netrec is a utility, reporting TCP, UDP and IP statistics from a running local or global zone. If the system deploys one or more zones and if all zones share same TCP/IP stack then you can simple use -s flag to report the numbers just once. The recorder outputs its data under RRD format and no delta are stored !

netrec used to observe TCP, UDP and IP counters. Recording points:

  • timestamp: seconds since Epoch
  • udpInDatagrams
  • udpInErrors
  • udpOutDatagrams
  • udpOutErrors
  • tcpActiveOpens
  • tcpPassiveOpens
  • tcpAttemptFails
  • tcpEstabResets
  • tcpCurrEstab
  • tcpOutSegs
  • tcpOutDataSegs
  • tcpOutDataBytes
  • tcpRetransSegs
  • tcpRetransBytes
  • tcpListenDrop
  • tcpListenDropQ0
  • tcpHalfOpenDrop
  • tcpOutSackRetrans
  • ipInHdrErrors
  • ipInAddrErrors
  • ipInCksumErrs
  • tcpInErrs
  • udpInCksumErrs

The output from netrec is displayed below: 

# netrec
global:1225354459:241756657:0:242661104:0:167385:475:54354:4477:17:209625:113456:407013:978714:913964897:0:0:0:435380:16:10
lobby:1225354459:241756657:0:242661104:0:167385:475:54354:4477:2:209627:113456:407013:978714:913964897:0:0:0:435380:16:10
wowza-test:1225354459:241756657:0:242661104:0:167385:475:54354:4477:0:209627:113456:407013:978714:913964897:0:0:0:435380:16:10
dss:1225354460:241756657:0:242661104:0:167385:475:54354:4477:0:209627:113456:407013:978714:913964897:0:0:0:435380:16:10
dss-test:1225354460:241756657:0:242661104:0:167385:475:54354:4477:0:209627:113456:407013:978714:913964897:0:0:0:435380:16:10

# netrec -s
global:1225354465:241756657:0:242661105:0:167386:475:54354:4477:16:209627:113457:40701586:978715:913964897:0:0:0:435380:161:10
lobby:1225354465:2
wowza-test:1225354466:0
dss:1225354466:0
dss-test:1225354466:0

Note: If you observe for shared TCP/IP stacks we use -s flag in order to skip reporting same data for all zones

procrec

Data Source /proc
API Sys::Statistics::Linux, Perl5

procrec records per process statistics: owner, state, nice, the priority of the process, the no. of light weight processes, the no. of open file descriptors, the no. of minor and major faults the process made and many other metrics. The recorder uses Sys::Statistics::Linux to fetch all metrics. The recorder has been modified to output its data as RRD and no delta values are stored, but rather raw data. This is a Linux only recorder.

procrec records per process metrics:

timestamp pid cmd owner state nice prior nlwp fds minflt majflt utime stime ttime
1313446375: 1: (init): root S: 0: 20: 1: 0: 0.00: 0.00: 0.00: 0.00: 0.00:

cutime cstime cpu size resident nswap cnswap syscr rchar read_bytes syscw wchar write_bytes
0.00: 0.00: 3: 24711168: 2347008: 0: 0: 0: 0: 0: 0: 0: 0

  • timestamp: seconds since Epoch
  • pid: process id
  • cmd: command of the process
  • owner: the owner name of the process
  • state: the status of the process
  • nice: the nice level of the process
  • prior: the priority of the process (+15)
  • nlwp: the no. of light weight processes by this process
  • fds: the no. of open file descriptors
  • minflt: the no. of minor faults the process made
  • majflt: the no. of major faults the process made
  • utime: the no. of jiffies proc have beed in user mode
  • stime: the no. of jiffies proc have beed in kernel mode
  • ttime: the no. of jiffies proc have beed (user + kernel)
  • cutime: the no. of jiffies proc waited for childs in user mode
  • cstime: the no. of jiffies proc waited for childs in kernel mode
  • cpu: the CPU number the process was last executed on
  • size: the total program size of the process
  • resident: no. of resident set size, the text, data and stack space
  • nswap: the size of swap space of the process
  • cnswap: the size of swap space of the childrens of the process
  • syscr: number of read syscalls
  • rchar: bytes read from storage (might have been from pagecache)
  • read_bytes: bytes really fetched from storage layer
  • syscw: number of write syscalls
  • wchar: bytes written
  • write_bytes: bytes sent to the storage layer
  • cmdline: command line of the process

Example running procrec on a Linux Ubuntu 11.04 desktop:

$ procrec -u parvu

1313492134:1830:(gnome-keyring-d):parvu:S:0:20:5:17:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:161607680:3629056:0:0:0:0:0:0:0:0
1313492134:1849:(sh):parvu:S:0:20:1:4:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:4321280:655360:0:0:0:0:0:0:0:0
1313492134:1883:(ssh-agent):parvu:S:0:20:1:0:0.00:0.00:0.00:0.00:0.00:0.00:0.00:3:12382208:299008:0:0:0:0:0:0:0:0
1313492134:1886:(dbus-launch):parvu:S:0:20:1:7:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:27168768:651264:0:0:0:0:0:0:0:0
1313492134:1887:(dbus-daemon):parvu:S:0:20:1:78:0.00:0.00:0.00:0.00:0.00:0.00:0.00:1:26292224:2170880:0:0:0:0:0:0:0:0
1313492134:1895:(xfconfd):parvu:S:0:20:1:9:0.00:0.00:0.00:0.00:0.00:0.00:0.00:3:34304000:2621440:0:0:0:0:0:0:0:0
1313492134:1901:(xscreensaver):parvu:S:0:20:1:9:0.00:0.00:0.00:0.00:0.00:0.00:0.00:1:69636096:3428352:0:0:0:0:0:0:0:0
1313492134:1903:(xfce4-session):parvu:S:0:20:2:20:0.00:0.00:0.00:0.00:0.00:0.00:0.00:1:196026368:8663040:0:0:0:0:0:0:0:0
1313492134:1910:(xfsettingsd):parvu:S:0:20:1:7:0.00:0.00:0.00:0.00:0.00:0.00:0.00:2:150966272:3792896:0:0:0:0:0:0:0:0
1313492134:1911:(Thunar):parvu:S:0:20:1:18:0.00:0.00:0.00:0.00:0.00:0.00:0.00:1:184537088:7827456:0:0:0:0:0:0:0:0
1313492134:1912:(xfwm4):parvu:S:0:20:2:14:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:396189696:16474112:0:0:10:0:0:10:11437:0
1313492134:1914:(gvfsd):parvu:S:0:20:1:9:0.00:0.00:0.00:0.00:0.00:0.00:0.00:2:57298944:2531328:0:0:0:0:0:0:0:0
1313492134:1918:(xfce4-panel):parvu:S:0:20:2:14:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:337727488:16723968:0:0:0:0:0:0:0:0
1313492134:1920:(gvfs-fuse-daemo):parvu:S:0:20:4:8:0.00:0.00:0.00:0.00:0.00:0.00:0.00:2:82952192:2756608:0:0:0:0:0:0:0:0
1313492134:1924:(xfdesktop):parvu:S:0:20:3:27:0.00:0.00:0.00:0.00:0.00:0.00:0.00:2:443224064:24432640:0:0:0:0:0:0:0:0
1313492134:1927:(xfce4-power-man):parvu:S:0:20:1:14:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:295698432:9666560:0:0:0:0:0:0:0:0
1313492134:1928:(xfce4-settings-):parvu:S:0:20:1:12:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:180498432:4845568:0:0:0:0:0:0:0:0
1313492134:1934:(xfce4-volumed):parvu:S:0:20:9:31:0.00:0.00:0.00:0.00:0.00:0.00:0.00:3:687271936:11718656:0:0:0:0:0:0:0:0
1313492134:1937:(polkit-gnome-au):parvu:S:0:20:2:12:0.00:0.00:0.00:0.00:0.00:0.00:0.00:1:168017920:7057408:0:0:0:0:0:0:0:0
1313492134:1939:(zeitgeist-datah):parvu:S:0:20:2:12:0.00:0.00:0.00:0.00:0.00:0.00:0.00:1:160595968:5746688:0:0:0:0:0:0:0:0
1313492134:1952:(xfce4-power-man):parvu:S:0:20:1:9:0.00:0.00:0.00:0.00:0.00:0.00:0.00:1:170340352:3526656:0:0:0:0:0:0:0:0
1313492134:1956:(zeitgeist-daemo):parvu:S:0:20:2:16:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:122855424:19021824:0:0:0:0:0:0:0:0
1313492134:1961:(applet.py):parvu:S:0:20:1:12:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:240422912:23592960:0:0:0:0:0:0:0:0
1313492134:1963:(update-notifier):parvu:S:0:20:2:23:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:388841472:15081472:0:0:0:0:0:0:0:0
1313492134:1967:(nm-applet):parvu:S:0:20:2:26:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:583507968:40296448:0:0:0:0:0:0:0:0
1313492134:1969:(gnome-volume-co):parvu:S:0:20:1:9:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:376619008:13705216:0:0:0:0:0:0:0:0
1313492134:1975:(xfce4-terminal):parvu:S:0:20:3:68:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:395702272:22183936:0:0:41:327:0:0:0:0
1313492134:1979:(gconfd-2):parvu:S:0:20:1:22:0.00:0.00:0.00:0.00:0.00:0.00:0.00:1:62750720:5926912:0:0:0:0:0:0:0:0
1313492134:1986:(panel-4-systray):parvu:S:0:20:1:7:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:174936064:7872512:0:0:0:0:0:0:0:0
1313492134:1988:(xfce4-indicator):parvu:S:0:20:3:14:0.00:0.00:0.00:0.00:0.00:0.00:0.00:1:408928256:29323264:0:0:0:0:0:0:0:0
1313492134:1989:(panel-9-xfsm-lo):parvu:S:0:20:2:13:0.00:0.00:0.00:0.00:0.00:0.00:0.00:2:389382144:13963264:0:0:0:0:0:0:0:0
1313492134:1992:(pulseaudio):parvu:S:-11:9:4:42:0.00:0.00:0.00:0.00:0.00:0.00:0.00:3:235442176:5623808:0:0:0:0:0:0:0:0
1313492134:1996:(panel-24-thunar):parvu:S:0:20:1:8:0.00:0.00:0.00:0.00:0.00:0.00:0.00:3:295505920:10731520:0:0:0:0:0:0:0:0
1313492134:2000:(gvfsd-trash):parvu:S:0:20:1:16:0.00:0.00:0.00:0.00:0.00:0.00:0.00:3:62128128:3432448:0:0:0:0:0:0:0:0
1313492134:2011:(gnome-pty-helpe):parvu:S:0:20:1:0:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:14962688:872448:0:0:0:0:0:0:0:0
1313492134:2012:(bash):parvu:S:0:20:1:4:0.00:0.00:0.00:0.00:0.00:0.00:0.00:1:33255424:10039296:0:0:0:0:0:0:0:0
1313492134:2014:(cat):parvu:S:0:20:1:3:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:11108352:598016:0:0:0:0:0:0:0:0
1313492134:2021:(bash):parvu:S:0:20:1:4:0.00:0.00:0.00:0.00:0.00:0.00:0.00:1:33251328:10043392:0:0:0:0:0:0:0:0
1313492134:2031:(indicator-sound):parvu:S:0:20:3:15:0.00:0.00:0.00:0.00:0.00:0.00:0.00:3:348069888:7061504:0:0:0:0:0:0:0:0
1313492134:2035:(indicator-appli):parvu:S:0:20:2:12:0.00:0.00:0.00:0.00:0.00:0.00:0.00:1:220979200:5345280:0:0:0:0:0:0:0:0
1313492134:2037:(indicator-messa):parvu:S:0:20:2:13:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:249167872:5459968:0:0:0:0:0:0:0:0
1313492134:2055:(gvfs-gdu-volume):parvu:S:0:20:1:10:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:73486336:3760128:0:0:0:0:0:0:0:0
1313492134:2090:(gvfs-gphoto2-vo):parvu:S:0:20:1:8:0.00:0.00:0.00:0.00:0.00:0.00:0.00:2:65081344:2727936:0:0:0:0:0:0:0:0
1313492134:2101:(gvfs-afc-volume):parvu:S:0:20:2:8:0.00:0.00:0.00:0.00:0.00:0.00:0.00:2:79208448:2695168:0:0:0:0:0:0:0:0
1313492134:2134:(zeitgeist-datah):parvu:Z:0:20:1:0:0.00:0.00:0.00:0.00:0.00:0.00:0.00:2:0:0:0:0:0:0:0:0:0:0
1313492134:2201:(ubuntuone-syncd):parvu:S:0:20:3:25:0.00:0.00:0.00:0.00:0.00:0.00:0.00:3:374575104:48336896:0:0:10:0:0:0:0:0
1313492134:2285:(gconf-helper):parvu:S:0:20:2:14:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:185712640:4083712:0:0:0:0:0:0:0:0
1313492134:2414:(bash):parvu:S:0:20:1:4:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:33259520:10174464:0:0:0:0:0:0:0:0
1313492134:2526:(fetchmail):parvu:S:0:20:1:3:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:55377920:3276800:0:0:0:0:0:0:0:0
1313492134:2564:(mutt):parvu:S:0:20:1:174:0.00:0.00:0.00:0.00:0.00:0.00:0.00:1:72994816:10772480:0:0:0:0:0:0:0:0
1313492134:2575:(firefox-bin):parvu:S:0:20:24:76:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:843001856:162553856:0:0:0:0:0:0:0:0
1313492134:2611:(bash):parvu:S:0:20:1:4:0.00:0.00:0.00:0.00:0.00:0.00:0.00:2:33251328:10035200:0:0:0:0:0:0:0:0
1313492134:2726:(bash):parvu:S:0:20:1:4:0.00:0.00:0.00:0.00:0.00:0.00:0.00:1:33296384:10256384:0:0:0:0:0:0:0:0
1313492134:2852:(plugin-containe):parvu:S:0:20:2:17:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:381333504:44982272:0:0:0:0:0:0:0:0
1313492134:3277:(ssh):parvu:S:0:20:1:7:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:41934848:2793472:0:0:0:0:0:0:0:0
1313492134:3429:(bash):parvu:S:0:20:1:4:0.00:0.00:0.00:0.00:0.00:0.00:0.00:1:33280000:10223616:0:0:0:0:0:0:0:0
1313492134:3910:(ssh):parvu:S:0:20:1:7:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:42074112:2793472:0:0:0:0:0:0:0:0
1313492134:3921:(bash):parvu:S:0:20:1:4:0.00:0.00:0.00:0.00:0.00:0.00:0.00:0:33251328:10162176:0:0:0:0:0:0:0:0
1313492134:4047:(procrec):parvu:R:0:20:1:5:5361.08:0.00:40.85:20.42:61.27:0.00:0.00:1:45101056:6549504:0:0:16553:1971:0:0:0:0

zonerec

Data Source prstat
API Perl5

zonerec is a simple script calling prstat utility to report zone utilisation in human readable format. This data needs to be parsed and prepared in RRD format. Future versions will include a new recorder which will output its data to RRD format.

zonerec used to observe CPU and Mem utilisation, as reported by prstat

The output from zonerec is displayed below: 

# zonerec 60

2008-10-30:10:32:12 - 1225355532
ZONEID    NPROC  SWAP   RSS MEMORY      TIME  CPU ZONE                        
     0      110 1025M 1050M    26%  17:22:56  11% global                      
     3       28  140M  199M   4.9%   3:24:32 0.1% wowza-test                  
     1       32   47M   46M   1.1%   0:00:45 0.0% lobby                       
     4       23   38M   37M   0.9%   0:00:38 0.0% dss                         
Total: 224 processes, 1064 lwps, load averages: 0.30, 0.30, 0.29

Zone: global
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
  7323 sparvu    236M  157M cpu0    14    0   4:30:39 7.0% firefox-bin/12
   640 sparvu    217M  169M sleep   59    0   4:46:25 1.8% Xorg/1
   698 sparvu     64M   62M sleep   59    0   6:45:11 1.7% prstat/1
  7343 sparvu    445M  307M sleep   57    4   0:33:13 0.6% java/76
   689 sparvu   4008K 2024K sleep   49    0   0:12:42 0.1% cpubar.x86/1
  7280 sparvu    258M  159M sleep   57    4   0:03:21 0.0% java/36
 17860 root     6848K 3704K cpu1    50    0   0:00:00 0.0% prstat/1
 17694 sparvu   5596K 1940K sleep   59    0   0:00:00 0.0% sshd/1
   605 root     3532K 1280K sleep   59    0   0:00:00 0.0% sshd/1
 17696 sparvu   1208K 1012K sleep   59    0   0:00:00 0.0% ksh/1
   139 root     2248K 1332K sleep   59    0   0:00:00 0.0% syseventd/14
   149 root     6108K 3500K sleep   59    0   0:00:00 0.0% devfsadm/9
 NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU                             
    56 sparvu    936M  969M    24%  17:19:14  11%
    41 root      299M  359M   8.8%   0:02:42 0.1%
     4 postgres   15M   16M   0.4%   0:00:59 0.0%
     1 lp       1016K 3012K   0.1%   0:00:00 0.0%
     1 smmsp    3712K 6040K   0.1%   0:00:00 0.0%
     6 daemon     38M   29M   0.7%   0:00:01 0.0%
Total: 109 processes, 390 lwps, load averages: 0.30, 0.30, 0.29

Zone: lobby
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
 17654 sparvu   5716K 1956K sleep   59    0   0:00:00 0.0% sshd/1
 17686 sparvu   4852K 3100K sleep   59    0   0:00:00 0.0% ssh/1
  1123 root     3532K 1340K sleep   59    0   0:00:00 0.0% sshd/1
  1120 root     2088K 1204K sleep   59    0   0:00:00 0.0% ttymon/1
  1164 root     9180K 5760K sleep   59    0   0:00:03 0.0% snmpd/1
  1149 root     1740K  660K sleep   59    0   0:00:00 0.0% smcboot/1
  1039 root     4064K 3016K sleep   59    0   0:00:04 0.0% inetd/4
  1047 root     1108K  640K sleep   59    0   0:00:00 0.0% utmpd/1
   774 root        0K    0K sleep   60    -   0:00:00 0.0% zsched/1
   986 daemon   6048K 3484K sleep   59    0   0:00:01 0.0% nfsmapid/3
   981 daemon   2440K 1064K sleep   59    0   0:00:00 0.0% rpcbind/1
   796 root       11M 9648K sleep   59    0   0:00:07 0.0% svc.startd/14
 NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU                             
     3 sparvu    936M  969M    24%   0:00:00 0.0%
     1 smmsp    3712K 6040K   0.1%   0:00:00 0.0%
     6 daemon     38M   29M   0.7%   0:00:01 0.0%
    22 root      299M  359M   8.8%   0:00:44 0.0%
Total: 32 processes, 119 lwps, load averages: 0.30, 0.30, 0.29


Zone: wowza-test
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
  1379 root      167M  114M sleep   49    0   3:23:47 0.1% java/225
  1350 root     1128K  820K sleep   59    0   0:00:00 0.0% sh/1
  1743 daemon   2104K 1328K sleep   59    0   0:00:00 0.0% lockd/2
  1673 root     2128K 1168K sleep   59    0   0:00:00 0.0% ttymon/1
  1872 root     1740K  672K sleep   59    0   0:00:00 0.0% smcboot/1
  1870 root     1740K  932K sleep   59    0   0:00:00 0.0% smcboot/1
  1388 root     5016K 2668K sleep   59    0   0:00:11 0.0% nscd/28
  1382 daemon   3912K 2028K sleep   59    0   0:00:00 0.0% kcfd/3
  1256 root       10M 8956K sleep   59    0   0:00:11 0.0% svc.configd/20
  1249 root     2160K 1196K sleep   59    0   0:00:00 0.0% init/1
   947 root        0K    0K sleep   60    -   0:00:00 0.0% zsched/1
  1254 root       10M 9100K sleep   59    0   0:00:07 0.0% svc.startd/12
  1378 root     1100K  784K sleep   49    0   0:00:00 0.0% startup.sh/1
 NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU                             
    22 root      299M  359M   8.8%   3:24:32 0.1%
     6 daemon     38M   29M   0.7%   0:00:00 0.0%
Total: 28 processes, 330 lwps, load averages: 0.30, 0.30, 0.29


Zone: dss
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
  2050 root     2088K 1208K sleep   59    0   0:00:00 0.0% ttymon/1
  1752 daemon   3912K 2020K sleep   59    0   0:00:00 0.0% kcfd/3
  2000 daemon   2104K 1328K sleep   59    0   0:00:00 0.0% lockd/2
  2090 root     1740K  936K sleep   59    0   0:00:00 0.0% smcboot/1
  1750 root     5544K 2640K sleep   59    0   0:00:12 0.0% nscd/27
  1975 daemon   2440K 1012K sleep   59    0   0:00:00 0.0% rpcbind/1
  2018 root     2104K 1160K sleep   59    0   0:00:00 0.0% ttymon/1
  1983 daemon   2456K 1596K sleep   59    0   0:00:00 0.0% statd/1
  2042 root     1108K  644K sleep   59    0   0:00:00 0.0% utmpd/1
  2091 root     1740K  672K sleep   59    0   0:00:00 0.0% smcboot/1
  2028 root     3536K 1592K sleep   59    0   0:00:00 0.0% syslogd/13
  1985 daemon   2124K 1320K sleep   59    0   0:00:00 0.0% nfs4cbd/2
  1294 root       11M 8756K sleep   59    0   0:00:07 0.0% svc.startd/13
  1972 root     2424K 1004K sleep   59    0   0:00:00 0.0% cron/1
 NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU                             
     6 daemon     38M   29M   0.7%   0:00:01 0.0%
    17 root      299M  359M   8.8%   0:00:37 0.0%
Total: 23 processes, 97 lwps, load averages: 0.30, 0.30, 0.29

corerec

Data Source SPARC CPU, coreadm
API Perl5

corerec is a utility using corestat, from Cooltools. The output is human readable format, it will require parsing and proper formating for RRD.

corerec used to observe core utilisation from a T1 or T2 processor. Since T1 and T2 have different registers for keeping track of the usage the corestat utility has to be different for each case: corestat.t1 used for T1 processors and corestat.t2 for T2. Recording points:

  • %Usr: percentage user time
  • %Sys: percentage sys time
  • %Usr+Sys: both Usr + Sys
  • Avg: core utilisation

The output from corerec is displayed below: 

For a T2 processor:
Core,Int-pipe     %Usr     %Sys     %Usr+Sys 
    0,0           0.05      0.37      0.42 
    0,1           0.06      0.07      0.13 
    1,0           0.04      0.08      0.12 
    1,1           0.01      0.05      0.06 
    2,0           0.09      0.11      0.20 
    2,1           0.01      0.06      0.07 
    3,0           0.02      0.15      0.17 
    3,1           0.01      0.05      0.05 
    4,0           0.01      0.13      0.14 
    4,1           0.01      0.04      0.05 
    5,0           0.07      0.10      0.16 
    5,1           0.01      0.45      0.46 
    6,0           0.02      0.10      0.12 
    6,1           0.01      0.06      0.07 
    7,0           0.03      0.12      0.14 
    7,1           0.01      0.05      0.06 
-------------     -----     -----    ------ 
    Avg           0.03      0.12      0.15

Important to note here is that utilisation for a T1 or T2 processor does not simple mean data from vmstat or mpstat alone. You have to use corerec in order to gather the correct utilisation figures. See below Ravindra Talashikar's notes about mpstat vmstat on T1 processors!

jvmrec

Data Source jstat
API Perl5

jvmrec is a utility based on jstat, part of JDK helping to extract Garbage Collection statistics from a running virtual machine. This recorder will loop over all running zones found on the system and will fetch each java process found and extract its GC numbers. The recorder outputs its data under RRD format.

jvmrec records the GC statistics useful to understand how your JVMs are running. Recording points:

  • zone.pid: name of the zone and process ID
  • timestamp: seconds since Epoch
  • S0%: Survivor s0 utilisation, percentage of the space's current capacity
  • S1%: Survivor s1 utilisation, percentage of the space's current capacity
  • Eden%: Eden space utilisation, percentage of the space's current capacity
  • Old%: Old space utilisation, percentage of the space's current capacity
  • Perm%: Permanent space utilisation, percentage of the space's current capacity
  • No.mGC: Number of young generation GC events
  • Time.mGC: Young generation garbage collection time, secs
  • No.MGC: Number of full GC events
  • Time.MGC: Full garbage collection time, secs
  • TotalGC: Total garbage collection time, secs

The output from jvmrec is displayed below:

zone.pid timestamp S0% S1% Eden% Old% Perm% No.mGC Time.mGC No.MGC Time.MGC Total GC
global.23699: 1225360607: 0.00: 29.51: 10.03: 10.07: 60.44: 9: 0.100: 1: 0.058: 0.158
global.23699: 1225360668: 0.00: 29.51: 12.76: 10.07: 60.44: 9: 0.100: 1: 0.058: 0.158

webrec

Data Source Target
API Apache HTTPClient, Java

webrec is a utility based on Apache HTTPClient, written in Java intended to be used to measure response times for different HTTP actions: POST, GET. Currently webrec supports only GET HTTP methods. Work in under way to expand this to POST methods and authentication.

webrec checks for a master configuration file, called: webrec.conf located under SDR/etc/ director, where SDR Recording module has been installed. It parses the configuration file: webrec.conf and it retrieves the values for timeout and the name definitions of each workload. For each workload it creates a thread and its starting to follow each URL recording the response times. WebRec supports Cookies as defined in RFC 2965

webrec operates against one or many workloads. Workload defines a sequence of URLs with the following properties:

  • name : name of the workload
  • delay : delay after each URL (in seconds)
  • interval: the interval to invoke workloads (in seconds)
  • timeout : determines the timeout until a connection is etablished. A value of zero means the timeout is not used. The default value is zero. (in milliseconds)

The output from webrec is displayed below:

timestamp transaction name response time (ms)
1273024907: main: 0.352
1273024912: recdesign: 0.355
1273024917: recorders: 0.381
1273024922: reporting: 0.318
1273024927: cpuplayer: 0.399
1273024932: relnotes: 1.174
1273024937: docs: 0.298
1273024942: faq: 0.366

The webrec.conf contains a single workload, called sdr with 8 transactions, each performed after Z=5 seconds. Delay parameter defines the number of seconds between each transaction.

<workload name="sdr" delay="5" interval="60" timeout="120">

    <transaction
        id="main"
        method="GET"
        url="http://www.systemdatarecorder.org/"
        value="xxx"/>

    <transaction
        id="recdesign"
        method="GET"
        url="http://www.systemdatarecorder.org/recording/recdesign.html"
        value="xxx"/>

    <transaction
        id="recorders"
        method="GET"
        url="http://www.systemdatarecorder.org/recording/recorders.html"
        value="xxx"/>

    <transaction
        id="reporting"
        method="GET"
        url="http://www.systemdatarecorder.org/reporting/"
        value="xxx"/>

    <transaction
        id="cpuplayer"
        method="GET"
        url="http://www.systemdatarecorder.org/cpuplayer/"
        value="xxx"/>

    <transaction
        id="relnotes"
        method="GET"
        url="http://www.systemdatarecorder.org/man/relnotes.html"
        value="xxx"/>

    <transaction
        id="docs"
        method="GET"
        url="http://www.systemdatarecorder.org/man/index.html"
        value="xxx"/>>

    <transaction
        id="faq"
        method="GET"
        url="http://www.systemdatarecorder.org/man/faq.html"
        value="xxx"/>

</workload>

At this moment webrec.conf must be manually written. Plans are to reuse the test plan generated by Apache jmeter to webrec, to easy the task of recording a series of transactions. You can however even now, record your workloads using jmeter and convert that to webrec or manually define the workload definition.