SDR - Frequently Asked Questions

This is the FAQ section for SDR

General

Why SDR ?
Recording
- License free
- Very simple
- Control over raw data in matter of minutes
- Raw data: time series, easy statistical modeling: trends, seasonal variations
- Very simple to change, add, remove whatever you need
- Easy to work with other modules, example PDQ
- Simple to educate your IT staff
Reporting
- License free
- Very simple
- Not an analytics package
- Statistical models: R and the analytical solver: PDQ will help to look over your data and predict the future
- You don't have to click, dozens of options, links to get what you need
- Should save your time in front of computer being simple and to the point
- Simple to educate your IT staff
But we have dozens of such systems already, example orca...
The main idea of SDR is to combine operating system metrics: CPU, Mem, Disk and Net I/O in terms of utilisation and queuing and couple these with RRD, R and PDQ for statistical analysis and prediction. SDR places an important role on the data collected, the raw data, which is simple stored on commodity disk drives: SATA, SAS without the need of a relational database management system.
How about HP SiteScope, IBM Tivoli ITCAM, etc...?
SDR tries to stay simple and follow GCaP methodologies and help you build a capacity plan for your IT infrastructure. Majority of the current commercial IT performance monitoring solutions are complex and large. They include application performance monitoring, end to end monitoring modules along with event alerting. These make them good for certain goals, bad for the other purposes. You need to dedicate a lot of time to learn and administer such systems. SDR focuses on: performance monitoring, analysis and forecasting. At the end you should have a simple Capacity Planning model for your site or application, with minimal time spent and investment.
How about: Zenoss, Ganglia, Munin or all the other open source efforts ?
They are all good for their job. Some started as network monitoring solutions, some measuring clusters and grids, majority showing plots about data all over. SDR tries to gather and have a logic between the data collected and follow a path towards capacity planning. As mentioned, SDR is a proof of concept built around GCaP methodologies.
Is SDR agentless or agent based system ?
Agent based system. Trying to avoid all marketing nonsense about this topic, agentless systems are much simple to setup, indeed, but they often are confusing: they don't list and define what probes are sent on each target and more complex because they use entirely the network to send-receive data. In majority of cases all agentless systems use Operating System utilities, like mpstat, vmstat to collect various metrics from each target. There is nothing innovative in such approach. SDR, on the other hand, has certain recorders which talk utilisation and saturation along with all the other OS metrics. As well SDR tries to stay simple and keep control over the raw data collected with minimal resources. Periodically the raw data is synced to a master site.

You can experiment with SDR recorders and use them directly as custom probes, if you wish, in a agentless system. SDR recorders have been designed to operate no matter what the reporting side is.
Which performance monitoring, analysis software should I use ?
It depends. There is no such thing as one tool for all jobs. Try to list and clear define what are your goals: raw data access, performance metrics of each monitored server, application performance monitoring, end to end response time, do you really need to have a capacity planning setup for your site, etc. Consult with your team members and your system or service manager and select a number of software to be evaluated. Select the best fits your organization.
Why do you always talk about GCaP ?
Because it is the human experience behind the performance analysis process not a specific software package! Majority of the IT companies simple buy software to replace people or their competence. That's wrong. And GCaP really helps you understand this and correct it. In addition, GCaP has all needed pieces to understand performance analysis and help you build a capacity plan for a site or a specific application without using any software package nor selling you one!
SDR, what R stands for: Recording or Reporting ?
System-Data-Recorder, so it means Recording. However you can have custom made packages for your installation where all recorded data can be displayed and lots of reports are available. The main idea of SDR is to combine the recording side with a light reporting module, based on RRD and Perl.
Is SDR smilar with Sensor Data Repository ?
No. Sensor Data Repository is used by ipmitool. When using ipmitool S-D-R means totally something else. SDR stands for System Data Recorder !
Can I install SDR under BSD or Linux based operating systems ?
SDR consists of two main things: Recording and Reporting. You can run one without the other, however many installations requires both. SDR recording at the moment requires Solaris due its tight integration with KSTAT interface. Work is under way to port SDR Recording module to FreeBSD/OpenBSD/NetBSD and RedHat. SDR Reporting can be installed under any POSIX based operating system: Solaris, FreeBSD or RedHat. Feel free to contribute the code for your operating system, if you would like to speed up the process of supporting more operating systems.
Why SDR has been developed initially under Solaris 10 ?
Beause of its merits regarding observability. In my opinion there is no other operating system on this planet so powerful and good documented as Solaris when talking about observability: KSTAT interface, operating system utilities, DTrace. Currently the main platform is OpenSolaris, Solaris 10 followed by FreeBSD and Linux.
Why there are 5 recorders or more ?
SDR recording part includes several recorders designed to collect data from a particular area of your systems: CPU, Zones, applications etc. Instead of having one, two general recorders I tried to design 5 main recorders which can be easily maintained and ported and others specialized for other purposes. Simplicity was the main criteria !
All recorders seem to be simple Perl or Ksh scripts, why ?
Simplicity was one of the main reasons behind. KSTAT interface in Solaris can be accessed via a Perl or C program. Brendan Gregg, the author of sysperfstat inspired me to keep using the same way, KSTAT scripts. When I was not able to obtain the information from KSTAT I used a simple Ksh script calling basic OS utilities. This last part needs improvement, example here zonerec, jvmrec. The main goal is to use as few utilities as possible and gather all data from OS interfaces.

Please note there are recorders, starting with SDR 0.73 which are native implementations, example nicrec.
SDR Team: who, when and how ?
We are a small group, majority of us having day jobs. We do most of the work during night, public holidays, summer or winter holidays. There is a full time developer implementing webrec, the response time analyzer in Java. We are able to answer any questions you have regarding SDR but currently there is no paid support for SDR. We will try to do our best and reply your emails in short time !
SDR Usage: commercial or non-commercial ?
www.systemdatarecorder.org is a research group focusing on: performance analysis, visualization and capacity planning. We are researching and building new tools which should help people and corporations in:
- analyzing and visualizing workloads
- analyzing infrastructure usage: web, app, database servers, storage, networks
- consolidate and save energy
systemdatarecorder.org is a not-for-profit organization. However, anyone interested in SDR can take our modules and sell and support them as they like. There are no restrictions regarding this !
We are a small/medium/large ISV and we would like to sell, support SDR. Is it possible ?
For sure. Visit the copyrights section and read carefully the documentation. You can do whatever you like with the recording or reporting modules. Please send us an email, if you found SDR useful. Thank you !
We are a small company and we would like to sell and support SDR. Is it possible ?
For sure. See above example.

Recording

Zonerec is based on prstat utility, why!?
To gather data from various Solaris zones, KSTAT interface should be used. Currently there is a open effort to improve this. Meanwhile prstat can be used to obtain data for each zone. Extended Process Accounting can as well be used to obtain information from each process running on the physical machine. However at this moment Im looking into new ways to improve this.
Sysrec does not correctly report memory utilisation
Make sure you use at least SDR 0.70 which includes updates related to sysrec and ZFS.
I see that nicrec is a binary file, why ?
Starting with SDR 0.73 certain recoders have been re-written in C. The main reasons: lower overhead, better support for hardware, portability. Nicrec is based on nicstat of Tim Cook.
Can I run SDR inside a VM guest ?
Yes, SDR supports VM guests. VirtualBox and VMware are the most common VM technologies and SDR supports them.
Can I run SDR inside a Solaris zone ?
Yes, SDR supports Solaris zones. Make sure you check your configuration: you can manage all zones from global level of the machine. If resource management is in use, then most likely you can consider using SDR inside a Solaris zone.
I see big timekeeping inaccuracies when running SDR inside a VM guest. Why ?
Make sure you read: "Timekeeping in VMware Virtual Machines" There are certain actions required to be set on the guest OS in order to minimize the chance of having inaccurate times. It has been observed if a guest OS CPU utilisation is 80-90% certain SDR recorders start loosing time. This is caused by the guest OS. Our internal testing has recorded such issues on a guest RHEL 5.4 32bit kernel 2.6.18-164. Solaris 10 10/09 s10x_u8wos_08a X86 kernel: 141445-09 seems to handle enough good situations like these.
What is webrec ?
SDR should be capable to analyze not only the operating system's resources but the applications, deployed on certain computer system infrastructure. Currently a large number of applications are deployed as web applications, using the HTTP protocol. A very clear requirement was to develop a light recorder to capture metrics from these applications. For start, the response time of one or many HTTP actions, was clear needed.

In order to gather the response times from a series of HTTP actions we have created webrec, a simple HTTP recorder based on Apache HTTPClient. The recorder is written in Java and is capable of recording multiple HTTP actions gathered as workloads. For each workload a thread is used. So you can use multiple workloads to record multiple HTTP actions of different parts of your site or sites !
Why should I care about webrec ?
If you care how well your HTTP site runs, or how efficient your web application has been developed you need to measure it and store its activity over a long period of time. webrec helps you in this respect measuring your application(s) and storing important metrics. Later you can conduct several types of analysis based on this set of data ! But first you need to have the data.
Does webrec support GET, POST or https, authentication ?
Currently webrec does support: GET and cookies. Work is under way to implement POST. Later on we are planning in adding support for https and authentication. This is based on SDR 0.73 release for 2010 year !

Reporting

Why can't I download a package for the reporting module ?
Not at this moment. The reporting side always differs from case to case. To make a generic package, which works all cases might be difficult. In near future a simple generic reporting package will be offered but it will require custom procedures to setup the reports, from case to case.
What operating systems the reporting module supports ?
UNIX: Solaris SPARC, x86, FreeBSD or Linux based operating systems.
Why can't I input a custom date under System Statistics - 7 days ?
Drill down report is currently being designed. You will be able to select a certain period of time, in addition to the preformat data, and data mine across all server's data and display all server metrics.
The reporting module seems a bunch of static images, why ?
SDR Reporting module is not a analytics package. It is not based on a relational database system nor has any substantial server side programming implemented.

The module includes the plots for different metrics and data gathered from all configured servers. In addition the reporting module displays as well other useful information: server characteristics, a per server stretch factor (a derivative metric out of the load average), a workload management part analyzing the applications running on that server and a prediction module.

There are several reasons why:
- Time: with minimal effort of clicks you should obtain the needed information, pre-formatted for you in matter of seconds. If you require certain custom reports, the reporting module has to be configure to do that.
- Simplicity: minimize the number of defects by having a complex server side software. Simple modify or customize the reporting module, based on your input.
- Efficiency: avoid complex Flash based solutions, which sometimes are eating your computer system resources: CPU, Network bandwidth.
- Education:Avoid spending lots of your cash in long and boring education packages, where you simple learn how to click between functions of a software package.
- Recovery:Long hours trying to restore a large database, where all your aggregated data is stored. Simple and easy recovery was another reason why to select Round Robin Database and a simple scripting language to do the job. In our lab to restore data for 1 server over a year took around approx. 10 minutes.
How is the reporting module developed: using open-source software ?
Yes. The reporting engine is using open-source software and all its components are as well available back as open-source software ! This way you can easily build your own reporting engine and integrate it as you like for your preferate operating system platform. However a prebuilt reporting package is not offered at this time, unless you are a registered user of SDR !
The reporting module is powered by a HTTP server. What server is that ?
Starting with SDR 0.73 the default webserver is Engine X, NGINX.
Can I use a different HTTP server instead the default one ?
Sure, you can use any HTTP server you like. Just make sure you install your HTTP server under /opt/sdr/report/ws or point your installation to SDR. Remember the default docroot is defined under /opt/sdr/report/docroot !
Can I compile my own version of Apache and use that for SDR ?
Sure, you can use Apache HTTP server for SDR. Compile and install Apache HTTP server and point your installation to SDR. See above example !
Why are you using NGINX HTTP server inside SDR Reporting module ?
Because of its merits: speed, simplicity, architecture and scalability.

Last updated: 2010-06-15

Back to main homepage