URL Monitor

The SiteScope URL Monitor is one of the most versatile and powerful Web monitoring tools available to Webmasters. Its core function is to attempt to reach a specified Web page to verify that it can be retrieved, but it can also be used to do the following:

Check secure pages using SSL, 128 bit SSL, and client certificates
Check for specific content on the retrieved Web page
Check the Web page for change
Check for specific error messages
Check the Web page for a value
Retrieve detailed download information
Check XML

When the URL Monitor retrieves a Web page, it retrieves the page's contents. A successful page retrieval assures you that your Web server is functioning properly. The URL Monitor does not automatically retrieve any objects linked from the page, such as images or frames. You can, however, instruct SiteScope to retrieve the images on the page by selecting the Retrieve Images or Retrieve Frames box located in the Advanced Options section of the Add Url Monitor Form.

In addition to retrieving specific Web pages, the URL Monitor can help you verify that CGI scripts and back-end databases are functioning properly. Just input the complete URL used to retrieve data from your database or trigger one of your CGI scripts, and the URL monitor will verify that the script generates a page and returns it to the user. For example, you can verify that your visitors are receiving a thank you page when they purchase something off of your site. The URL monitor's string matching capability even allows you to verify that the contents of the page are correct.

Each time the URL Monitor runs, it returns a reading or status and writes it in the monitoring log file. It also writes in the log file the total time it takes to receive the designated document. This status value is also displayed in the SiteScope Monitor tables and is included as part of alert messages sent via e-mail.

Usage Guidelines

The SiteScope URL Monitors provide you with end-to-end verification that your Web server is running, serving pages correctly, and doing so in a timely manner. Because it tests end-to-end, it is also able to determine whether back-end databases are available, verify the content of dynamically generated pages, check for changed content, and look for specific values from a page.

What to monitor

You can create URL monitors to watch pages that are critical to your Web site (such as your home page), pages that are generated dynamically, and pages that depend upon other applications to work correctly (such as pages that utilize a back-end database). Your goal is to monitor a sampling of every type of page you serve to ensure that everything is working properly. There is no need to verify that every page of a particular type is working correctly -- one or two will do.

When you are choosing which pages to monitor, select pages with the lowest overhead. For example, if you have several page that are generated by another application, monitor the shortest one with the fewest graphics. This will put less load on your server while still providing you with the information you need about system availability.

About scheduling this monitor

Each URL Monitor puts no more load on your server than someone accessing your site and retrieving a page, so in most cases you can schedule them as closely together as you want. Keep in mind that the length of time between each run of a monitor is equal to the amount of time that can elapse before you are notified of a possible problem.

A common strategy is to schedule monitors for very critical pages to run every 1 to 2 minutes, and then schedule monitors for less critical pages to run only every 10 minutes or so. Using this strategy, you will be notified immediately if a critical page goes down or if the entire Web site goes down, but you will not have an excessive number of monitors running all the time.

Status

The status reading shows the most recent result for the monitor. This status value is displayed in the URL Group table within SiteScope. It is also recorded in the SiteScope log files, e-mail alert messages, and can be transmitted as a pager alert. The possible status values are:

OK
unknown host name
unable to reach server
unable to connect to server
timed out reading
content match error
document moved
unauthorized
forbidden
not found
proxy authentication required
server error
not implemented
server busy

The status is logged as either good, warning, or error. A warning status or error status is returned if the current value of the monitor is a condition that you have defined as other than OK.

Completing the URL Monitor Form

To display the URL Monitor Form, either click the Edit link for an existing URL Monitor in a monitor table, or click the add a Monitor link on a group's detail page and click the "Add URL Monitor link.

Complete the items on the URL Monitor form as follows. When the required items are complete, click the Add Monitor button.

URL: Enter the URL that you want to monitor (for example, http://demo.thiscompany.com).
Note:for HTTPS monitoring (secure HTTP):; If the URL starts with HTTPS, then a secure connection will be made using SSL (for example, https://www.thiscompany.com). SiteScope uses Java SSL libraries for HTTPS monitoring. For SiteScope on NT, SiteScope uses the SSL library from Internet Explorer if you select the NT Challenge Response option. To monitor using SSL and NT Challenge Response, installing Internet Explorer 4.0 or later is recommended.
Match Content: Enter a string of text to check for in the returned page or frameset. If the text is not contained in the page, the monitor will display "no match on content". The search is case sensitive. Remember that HTML tags are part of a text document, so include the HTML tags if they are part of the text you are searching for (for example, " Hello World"). This works for XML pages as well. You may also perform a Perl regular expression match by enclosing the string in forward slashes, with an "i" after the trailing slash indicating case-insensitive matching. (for example, "/href=Doc\d+\.html/" or "/href=doc\d+\.html/i"). If you want a particular piece of text to be saved and displayed as part of the status, use parentheses in a Perl regular expression. For example /Temperature: (\d+). This would return the temperature as it appears on the page and this could be used when setting an Error if or Warning if threshold.
Update every: Select how often the monitor should check this URL. The default interval is to run or update the monitor once every 10 minutes. Use the drop-down list to the right of the text box to specify another update interval in increments of seconds, minutes, hours, or days. The update interval must be 15 seconds or longer.
Title: Enter a title text for this monitor. This text is displayed in the group detail page, in report titles, and other places in the SiteScope interface. If you do not enter a title text, SiteScope will create a title based on the host, server, or URL being monitored.

Advanced Options

The Advanced Options section presents a number of ways to customize monitor behavior and display. Use this section to customize error and warning thresholds, disable the monitor, set monitor-to-monitor dependencies, customize display options, and enter other monitor specific settings required for special infrastructure environments. The options for this monitor type are described below. Complete the entries as needed and click the Add or Update button to save the settings.

Disable

Check this box to temporarily disable this monitor and any associated alerts. To enable the monitor again, clear the box.

Timeout

The number of seconds that the URL monitor should wait for a page to complete downloading before timing-out. Once this time period passes, the URL monitor will log an error and report an error status. If you have checked the Retrieve Frames or Retrieve Images option, SiteScope will wait for these items to be retrieved before considering the page to be fully downloaded.

HTTP Proxy

Optionally, a proxy server can be used to access the URL. Enter the domain name and port of an HTTP Proxy Server.

Retrieve Images

Check this box if you want the status and response time statistics to include the retrieval times for all of the embedded images in the page. Embedded images include those referenced by "IMG" and "INPUT TYPE=IMAGE" HTML tags. Images that appear more than once in a page are only retrieved once.

Note: If the Retrieve Images option is checked, each image referenced by the target URL will contribute to the download time. However, if a image times out during the download process or has a problem during the download, that time will not be added to the total download time.

Retrieve Frames

Check this box if you want SiteScope to retrieve the all frames references in a frameset and count their retrieval time in the total time to download this page. If Retrieve Images is also checked, SiteScope will attempt to retrieve all images in all frames.

Note: If the Retrieve Frames option is checked, each frame referenced by the target URL will contribute to the download time. However, if a frame times out during the download process or has a problem during the download, that time will not be added to the total download time.

Error If Match

Enter a string of text to check for in the returned page or frameset. If the text is contained in the page, the monitor will indicate an error condition. The search is case sensitive. Remember that HTML tags are part of a text document, so include the HTML tags if they are part of the text you are searching for (for example, " Error Message"). You may also perform a Perl regular expression match by enclosing the string in forward slashes, with an "i" after the trailing slash indicating case-insensitive matching. (for example, /href=Doc\d+\.html/" or "/href=doc\d+\.html/i).

Check for Content Changes

Unless this is set to "no content checking" (the default) SiteScope will record a checksum of the document the first time the monitor runs and then does a checksum comparison each subsequent time it runs. If the checksum changes, the monitor will have a status of "content changed error" and go into error. If you want to check for content changes, you will usually want to use "compare to saved contents".

The options for this setting are:

no content checking - (default) SiteScope does not check for content changes
compare to last contents - The new checksum will be recorded as the default after the initial error "content changed error" occurs, so the monitor will return to OK until the checksum changes again
compare to saved contents - The checksum is a snapshot of a given page (retrieved either during the initial or a specific run of the monitor). If the contents change, the monitor will get a "content changed error" and will stay in error until the contents return to the original contents, or the snapshot is update by resetting the saved contents
reset saved contents - Takes a new snapshot of the page and saves the resulting checksum on the first monitor run after this option is chosen. After taking the snapshot, the monitor will revert to "compare to saved contents" mode.

Authorization User Name

If the URL specified requires a name and password for access, enter the user name in this box.

Authorization Password

If the URL specified requires a name and password for access, enter the password in this box.

NT Challenge Response

Check this box if you want SiteScope to use Window's NT Challenge Response authorization when retrieving this Web page.

Proxy Server User Name

If the proxy server requires a name and password to access the URL, enter the name here. Technical note: your proxy server must support Proxy-Authenticate for these options to function.

Proxy Server Password

If the proxy server requires a name and password to access the URL, enter the password here. Technical note: your proxy server must support Proxy-Authenticate for these options to function.

POST Data

If the URL is for a POST request, enter the post variables, one per line as name=value pairs. This option is used to verify that a form is working correctly by performing the same request that occurs when a user submits a form. See also the Match Content item for a way to verify that the correct form response was received. If this item is blank, a GET request is performed.

Advanced Option The POST Data can be used to send cookie data. To send cookies with the request use the format "Set-cookie: cookieName=cookieValue".
To change the content type of a post, use the format "Content-Type: application/my-format". To hide values in the POST data, add a line like:

_private=_name=mysecret _value=rosebud
_private=_name=mypassword _privateValue=sesame

and then use the following form in the POST Data

s|username=$private-mysecret$|
s|password=$private-mypassword$|

and SiteScope will substitute the values from the master.config into the POST Data.

Error If Redirected

Check this box if you want SiteScope to notify you if a URL is redirected. Normally, SiteScope follows redirects without reporting an error.

Show Detailed Measurement

Check this box if you want SiteScope to record a detailed break down of the process times involved in retrieving the requested URL. This includes DNS lookup, connect time, HTTP server response time, described as follows:

DNS time - The time it takes to send a name resolution request to your DNS server until you get a reply.
Connection time - The time it takes to establish a TCP/IP/Socket connection to the Web server.
Response time - The time after the request is sent until the first byte (rather first buffer full) of the page comes back.
Download time - The time it takes to download the entire page.

Baseline Interval

Enter the number of monitor runs to be averaged for use as a Rolling Baseline. Rolling baselines are calculated for an interval equal to time to complete the number of monitor runs entered here. See more information.

Verify Error

Check this box if you want SiteScope to automatically run this monitor again if it detects an error. When an error is detected, the monitor will immediately be scheduled to run again once.

Note: In order to change the run frequency of this monitor when an error is detected, use the Update every (on errors) option below.

Note: The status returned by the Verify Error run of the monitor will replace the status of the originally scheduled run that detected an error. This may cause the loss of important performance data if the data from the verify run is different than the initial error status.

Warning: Use of this option across many monitor instances may result in significant monitoring delays in the case that multiple monitors are rescheduled to verify errors at the same time.

Update Every (on error)

Use this option to set a new monitoring interval for monitors that have registered an error condition. For example, you may want SiteScope to monitor this item every 10 minutes normally, but as often as every 2 minutes if an error has been detected. Note that this increased scheduling will also affect the number of alerts generated by this monitor.

Schedule

By default, SiteScope monitors are enabled every day of the week. You may, however, schedule your monitors to run only on certain days or on a fixed schedule. Click the Edit schedule link to create or edit a monitor schedule. For more information about working with monitor schedules, see the section on Schedule Preferences for Monitoring.

Monitor Description

Enter additional information about this monitor. The Monitor Description can include HTML tags such as the <HR>, and tags to control display format and style. The description will appear on the Monitor Detail page.

Report Description

Enter an optional description for this monitor that will make it easier to understand what the monitor does. For example, network traffic or main server response time. This description will be displayed on with each bar chart and graph in Management Reports and appended to the tool-tip displayed when you pass the mouse cursor over the status icon for this monitor on the monitor detail page.

Depends Condition

If you choose to make the running of this monitor dependent on the status of another monitor, select the status condition that the other monitor or monitor group should have in order for the current monitor to run normally. The current monitor will be run normally as long as the monitor on which it depends reports the condition selected in this option.

List Order

By default, new monitors are listed last on the Monitor Detail page. You may use this drop-down list to choose a different placement for this monitor.

Setting URL Monitor Status Thresholds

SiteScope URL monitor types allow you to set threshold conditions based on part or all of a Web page retrieval or transaction to determine the status reported by each monitor. Monitor status can be set based on the roundtrip, DNS, connect, or response times needed to negotiate a Web request or transaction.

The roundtrip time is one of the most commonly used metrics for URL monitoring. Roundtrip time includes the time to complete the following:

Name lookup (DNS)
Connect to the server socket
Send the HTTP request
Download the entire page

If the server hosting SiteScope is heavily loaded, either with other applications or with a high SiteScope monitoring load, then it can contribute to slower times, especially for HTTPS URLs, which are more CPU intensive. Usually though, the time waiting for the network is a much greater factor than the CPU usage.

Set the monitor status thresholds for the URL Monitor as described below.

Error if

By default, SiteScope generates an error if the returned HTTP status is anything other than 200 ("OK"), which indicates a successful retrieval. You can choose to have SiteScope report an error status based on any of the following measurements:

round trip time - the total time for the entire request, in milliseconds
DNS time - the amount of time to translate the host name to an IP address, in milliseconds
connect time - the amount of time to make the connection, in milliseconds
response time - the amount of time before the first response was received, in milliseconds
download time - the amount of time to receive the page contents, in milliseconds
age -- the amount of time between the current time and the last-modified time for the page, in seconds
content match
total errors
overall status

Choose a comparison operator from the drop-down list, and enter a value for the comparison in the text box.

The URL Monitor follows HTTP redirect codes (301 and 302) to retrieve the actual page before returning the status of the URL retrieval. SiteScope will show a redirect error only if the redirects are more than 10 levels deep - this prevents infinite redirects from being followed, or if the Error On Redirect check box is selected.

Warning if

By default, SiteScope generates a warning if Check Images or Check Frames is chosen and there was a problem retrieving one of the images or frames. You may choose to have the monitor report a warning status based on any of the measurement options listed under Error if. Choose a comparison operator, and enter a value for the comparison to generate a warning.

Good if

By default, SiteScope reports an OK status if the URL returns an HTTP status of 200 ("OK"). You may also choose to have SiteScope base an OK status on any of the measurement options listed under Error if. Enter the value that should generate an OK status.