SiteScope User's Guide


Script Monitor

The SiteScope Script Monitor runs an external command and reports the result. It is an easy way to integrate existing system management scripts into the SiteScope environment. The Script Monitor can be tailored to run scripts at regular intervals. In addition to reporting the command result, the Script Monitor can also parse and report a specific value from the command output.

Each time the Script Monitor runs, it returns a status and writes it into the monitoring log file. It also reports a command result, a value, and the time it took to run the command.

Usage Guidelines

One of the primary reasons for using the Script Monitor is to integrate an existing script that you use to do a particular system management function into SiteScope. For example, if you have a script that runs a diagnostic on an application and returns a 0 reading if everything's OK, you could create a script monitor that runs this script and recognizes any exit value other than 0 as an error. Then you could create an alert which would e-mail or page you in the event that this monitor was in error.

The Script Monitor can be used to run shell commands or other scripts on the machine where SiteScope is running or it can run a script that is stored on a remote machine. The following is an overview of the possible script execution options and requirements for the SiteScope Script Monitor:

Script Option

Description

Local Script A file stored and executed on the SiteScope machine. The file should be stored in the <SiteScope install path>/SiteScope/scripts directory.
Remote Script A remote script file (UNIX and NT-NT SSH ONLY) in a scripts subdirectory in the home directory of the account SiteScope uses to access the remote server. For example, home/sitescope/scripts. The remote scripts must include an echo construct to echo script results and exit codes back to SiteScope (see the Return Status Example section below). The monitor may fail if the appropriate exit code is not echoed back to SiteScope.
Remote Command A script file containing a single command stored locally in the <SiteScope install path>/SiteScope/scripts.remote directory. This script file is used to run a command on a remote server. The command may be used to execute a remote script file that performs multiple functions.

Note: For SiteScope on Linux, the script itself must have a shell invocation line as the very first line of the script. This applies to scripts that you are trying to run locally on the SiteScope machine. For example, the first line of the script should include something like #!/bin/sh or #!/usr/local/bin/perl. If the shell invocation line is not found then the exec() call will return with a -1 exit status. This is a limitation of the Java Runtime in JRE prior to release 1.4. This has been fixed in the 1.4 JRE from Sun which is shipped with SiteScope version 7.8 and later.

Scheduling Script monitors is dependent upon the script that you want SiteScope to run. You can use the scheduling option to have SiteScope run scripts at different intervals throughout the week.

Status

The command result is the exit value returned by running the command. This works for local UNIX scripts, but does not work for remote UNIX scripts, or Win NT batch files. Win NT batch file (*.bat) exit codes are not passed out of the command interpreter, and remote UNIX script exit codes are not passed back through the remote connection. See the example below for a way to receive information from the script.

Completing the Script Monitor Form

To display the Script Monitor Form, either click the Edit link for an existing Script Monitor in a monitor table, or click the Add a new Monitor to this Group link on a group's detail page and click the Add Script Monitor link.

Complete the items on the Script Monitor form as follows. When the required items are complete, click the Add Monitor button.

Script
Enter the name of the script to run. For security reasons, only scripts placed into the <SiteScope install path>/SiteScope/scripts directory may be used. In that directory, there are several examples scripts with comments describing each one.

If you choose "USE COMMAND", your must also specify a USE COMMAND script file name in the Advanced Options section below. SiteScope will send the command or commands found in the USE COMMAND script file to be executes as a command line on the remote UNIX Machine. Script files for the USE COMMAND option must be created in the <SiteScope install path>/SiteScope/scripts.remote directory.
For Example: Create a file named test.sh and save it in the <SiteScope install path>/SiteScope/scripts.remote directory. Edit test.sh to include the command syntax

ps -ef;echo "all done"

as the content of the file. Then create a Script monitor with the USE COMMAND option selected, select a remote UNIX machine, and select test.sh as the USE COMMAND script to run.

Server
By default, SiteScope will execute script files that are stored locally on the SiteScope machine in the <SiteScope install path>/SiteScope/scripts directory. You can have SiteScope execute a script that is stored on a remote machine by selecting the remote machine using the Choose Server screen. Click the Choose Server link to display the Choose Server screen. Use the drop-down list to select a remote machine . If the remote machine you are looking for does not apear in the list, you will need to create a connection profile using the Remote UNIX Preferences page.

Parameters
Use this text box to specify any additional parameters to pass to the script. Optionally, you can use a regular expression or one of SiteScope's date variables to insert date and time into the parameters box. For example, s/$month$ $day$ $year$/ will pass the current month, day and year to the script.

Note: For security reasons the following characters are not allowed to be passed to scripts by SiteScope:

` (apostrophe)
; (semicolon)
& (ampersand)
| (vertical pipe)
< (less than)
> (greater than)

Update every
Select how often the monitor should run this Script. How often the monitor should be run depends on what action it performs or what data it returns. The default interval is to run or update the monitor once every 10 minutes. Use the drop-down list to the right of the text box to specify another update interval in increments of seconds, minutes, hours, or days. The update interval must be 15 seconds or longer.

Title
Enter a title text for this monitor. This text is displayed in the group detail page, in report titles, and other places in the SiteScope interface. If you do not enter a title text, SiteScope will create a title based on the host, server, or URL being monitored.

Advanced Options

The Advanced Options section presents a number of ways to customize monitor behavior and display. Use this section to customize error and warning thresholds, disable the monitor, set monitor-to-monitor dependencies, customize display options, and enter other monitor specific settings required for special infrastructure environments. The options for this monitor type are described below. Complete the entries as needed and click the Add or Update button to save the settings.

USE COMMAND Script File
If you have selected the USE COMMAND as the Script option above and a remote machine as the Server, select the script file that contains the commands that SiteScope should send to the remote machine. You can save one or more commands in the text script file and save the file in the <SiteScope install path>/SiteScope/scripts.remote directory. SiteScope will open this file and run the command at the command line of the remote server chosen in the "choose server" option above. You can then use the Match Expression option to parse the output of the command and display valuable information.

The USE COMMAND script can make use of positional parameters such as $1, $2 (or alternatively %1, %2), and so forth, inside the script. Enter the parameters you want SiteScope to pass to the script in the Parameters box provided above.

Note: You can use one or more commands per USE COMMAND script file. DO NOT include any carriage returns or any command that would normally discontinue script processing (for example, do not use the exit command).

Disable
Check this box to temporarily disable this monitor and any associated alerts. To enable the monitor again, clear the box.

Match Expression
To retrieve a value from the script output, enter a Perl regular expression in this box. For example, the expression: /(\d+)/ will match one or more digits returned by the script. The retrieved value can be used to set the error or warning status of the monitor and to trigger alerts. SiteScope will check up to four values returned. If this item is left blank, no value will be retrieved from the script.

Verify Error
Check this box if you want SiteScope to automatically run this monitor again if it detects an error. When an error is detected, the monitor will immediately be scheduled to run again once.

Note: In order to change the run frequency of this monitor when an error is detected, use the Update every (on errors) option below.

Note: The status returned by the Verify Error run of the monitor will replace the status of the originally scheduled run that detected an error. This may cause the loss of important performance data if the data from the verify run is different than the initial error status.

Warning: Use of this option across many monitor instances may result in significant monitoring delays in the case that multiple monitors are rescheduled to verify errors at the same time.

Update Every (on error)
You use this option to set a new monitoring interval for monitors that have registered an error condition. For example, you may want SiteScope to monitor this item every 10 minutes normally, but as often as every 2 minutes if an error has been detected. Note that this increased scheduling will also affect the number of alerts generated by this monitor.

Schedule
By default, SiteScope monitors are enabled every day of the week. You may, however, schedule your monitors to run only on certain days or on a fixed schedule. Click the Edit schedule link to create or edit a monitor schedule. For more information about working with monitor schedules, see the section on Schedule Preferences for Monitoring.

Monitor Description
Enter additional information to display about this monitor. The Monitor Description can include HTML tags such as the <BR> <HR>, and <B> tags to control display format and style. The description will appear on the Monitor Detail page.

Report Description
Enter an optional description for this monitor that will make it easier to understand what the monitor does. For example, network traffic or main server response time. This description will be displayed on with each bar chart and graph in Management Reports and appended to the tool-tip displayed when you pass the mouse cursor over the status icon for this monitor on the monitor detail page.

Depends On
To make the running of this monitor dependent on the status of another monitor or monitor group, use the drop-down list to select the monitor on which this monitor is dependent. Select None to remove any dependency.

Depends Condition
If you choose to make the running of this monitor dependent on the status of another monitor, select the status condition that the other monitor or monitor group should have in order for the current monitor to run normally. The current monitor will be run normally as long as the monitor on which it depends reports the condition selected in this option.

List Order
By default, new monitors are listed last on the Monitor Detail page. You may use this drop-down list to choose a different placement for this monitor.

Error if
By default, SiteScope generates an error if the script returns an error as the exit status. You may choose to generate an error based on the exit status, any one of four matched script values, or the time it takes to run the script. Select a comparison value or enter a value in the box provided. Values may be numeric or can be text strings returned by the script. To use a text value, enclose the text string in single quotes: 'text'. Use the comparison operator list to specify an error threshold such as: >= (greater than or equal to), != (not equal to), or < (less than).

Warning if
By default, SiteScope does not generate warnings for Script monitors. You may choose to generate a warning based on the exit status, a text message, the script value, or the time it takes to run the script.

Good if
SiteScope reports a good status if the exit status of the script equals 0, but you can change this to be based upon another value or round-trip time.

Script Return Status Example

In order to get around the fact that exit codes that are not returned to SiteScope after execution of Win NT batch files or UNIX scripts executed on remote servers, we recommend including an echo to standard out of a return value. In the case of Win NT-to-NT remote scripts (using Secure Shell), the remote script MUST echo end script when the script has terminated. Other returned values can then be matched in the Script monitor using a regular expression in the Match Expression box.

The following is an example script outline based on a UNIX shell script.

In the script that will run on a remote server include echo commands that represent the different logical paths that might be followed:

#!/bin/sh
...(script commands and logic here)...
echo "Return Code: 1" (indicating the script failed to complete execution)
...(more script commands and logic here)...
echo "Return Code: 0" (the end of the script, indicating the script completed successfully)

Under the Advanced Options in the Script Monitor set up page, create a Match Expression using the following regular expression pattern:

/Return Code: (\d+)/
    

Then set the Error, Warning, and Good thresholds for the monitor as follows:

Error if value > 0
Warning if value == 'n/a'
Good if value == 0

With this set up, if the echoed Return Code value is greater than 0, it signals that the script did not execute correctly. If the script does not run properly, meaning that no Return Code echo command in the script is executed, then a warning condition occurs (for example, there will not be a match for the Match Expression which will return a 'n/a'). If the script echoes the Return Code of 0, then a good condition is detected. In this case the monitor status shown on the monitor detail page will display "matched 0" if the script executed successfully.

Setting a Timeout Value for Script Execution

You can set a timeout value for the Script Monitor for SiteScope running on Windows. The timeout value is the total time, in milliseconds, that SiteScope should wait for a successful run of the script. You can use this option to have SiteScope run the monitor but kill the script execution if a script exit code is not detected within the timeout period.

The requirements and limitations of this option are:

  • is only available with SiteScope for Windows
  • can only be used with scripts stored and invoked on the local SiteScope server (that is, where the Server setting for the Script Monitor is this server or localhost)
  • the timeout setting value is expressed in milliseconds
  • only applies to Script Monitors

Two methods exist for applying a timeout setting to Script monitors. One applies the setting as a property to an individual monitor. The second method adds the setting to groups, subgroups, or the entire SiteScope installation. The procedures for both are described below.

To set a Timeout Value for Individual Script Monitors

  1. Stop the SiteScope service.
  2. Using a text editor, open the SiteScope group file containing the monitor frame for the Script Monitor to which you want to apply the timeout setting.
  3. Inside the Script Monitor frame (delimited by the # sign), insert a line and add the timeout setting as _timeout=mstime where mstime is replaced with the time in milliseconds.
  4. Save the group file. Note: Do not add blank lines, leading or trailing spaces to any record in the group file.
  5. Restart the SiteScope service.

To set a Timeout Value for Multiple Script Monitors

Script timeout option can be set globally or in a group-wise manner by the following steps:

  1. Stop the SiteScope service.
  2. Using a text editor, open the SiteScope group file containing one or more Script monitors to which you want to apply the timeout setting. Alternately, you can add the setting to the SiteScope/groups/master.config file.
  3. Inside the group file frame a the top of the file before the first # symbol, insert a line and add the timeout setting as _scriptMonitorTimeout=mstime where mstime is replaced with the time in milliseconds.
  4. Save the group file. Note: Do not add blank lines, leading or trailing spaces to any record in the group file or master.config file.
  5. Restart the SiteScope service.