URL List Monitor

The SiteScope URL List Monitor is used to check a large list of URLs. This monitor is commonly used by Web hosting providers to measure the availability and performance of their customer's Web sites.

A URL List is specified by giving a filename containing the list of URLs to check. The URLs that you want to monitor are saved in a plain text file. There is virtually no limit to the number that you can list though the run interval selected for the monitor may require that the number of URL's be limited. For each URL included in the URL list file, the monitor retrieves the contents of the URL or the server response to the request.

Each time the URL List Monitor runs, it returns the number of errors, if any, and writes it into the monitoring log file. It also writes the total number of URLs checked and the average time, in milliseconds, to retrieve each URL.

Usage Guidelines

You can use the URL List Monitor to check a list of URLs without having to create a separate URL monitor for each one. For example, this is useful if you host several Web sites and simply want to ensure that they are each serving pages properly. The URL List Monitor is not used to confirm links between pages (see the Link Check Monitor) or other Web transaction processes (see the URL Sequence Monitor).

What to monitor

The URL List Monitor is useful for monitoring any set of URL's that you simply want to make sure are available over the network.

About scheduling this monitor

This is strictly dependent upon how often to want to check to see if the URLs are working. Once an hour is common, but you can schedule it to run more often.

There are a few factors that affect how long it takes the URL List Monitor to complete a run: 1) number of URLs in the list, 2) URL retrieval time, and 3) the number of threads used. In some cases this may lead to the monitor not running as expected. As an example, assume you have a list of 200 URLs that you want to monitor every 10 minutes, but, due to Internet traffic, SiteScope is not able to complete checking all of the 200 URLs in that amount of time. The next time the monitor was scheduled to run, SiteScope would see that it did not complete the previous run and would wait for another 10 minutes before trying again.

If this happens once in awhile, it is probably not a problem, but if it happens more often there are three things you can do to resolve the issue.

The most obvious option is to schedule the monitor to run less frequently, but if that conflicts with some other objective, go to options 2 and 3.
The second thing you can do is reduce the pause interval set under the Advanced Options. This will minimize the time it takes for the monitor to retrieve all of the URLs.
The third option (which you can use in conjunction with number 2) is to increase the number of threads that SiteScope can use when checking the URLs. The more threads, the quicker SiteScope can check them. Of course, this will put a heavier load on your system, so you have to find a happy medium.

Ideally, you want SiteScope to have just completed checking the URLs in the list when it is time to start checking again. This would indicate that the load was evenly balanced. It may take some tweaking to get it just right.

Completing the URL List Monitor Form

To display the URL List Monitor Form, either click the Edit link for an existing URL List Monitor in a monitor table, or click the add a Monitor link on a group's detail page and click the Add URL List Monitor link.

Complete the items on the URL List Monitor form as follows. When the required items are complete, click the Add Monitor button.

URL List Name

Enter the path name for the file containing the list of URLs to be monitored. This file should be a plain text file and contain only one URL per line of text as shown here:

http://www.website.com/index.html http://www.website.com/main/customer/order.html http://www.website.net/default.htm http://www.Web pages.com/tech/support/ws/intro.html

Update every

Select how often the monitor should check this URL. The default interval is to run or update the monitor once every 10 minutes. Use the drop-down list to the right of the text box to specify another update interval in increments of seconds, minutes, hours, or days. The update interval must be 15 seconds or longer.

Title

Enter a title text for this monitor. This text is displayed in the group detail page, in report titles, and other places in the SiteScope interface. If you do not enter a title text, SiteScope will create a title based on the host, server, or URL being monitored.

Advanced Options

The Advanced Options section presents a number of ways to customize monitor behavior and display. Use this section to customize error and warning thresholds, disable the monitor, set monitor-to-monitor dependencies, customize display options, and enter other monitor specific settings required for special infrastructure environments. The options for this monitor type are described below. Complete the entries as needed and click the Add or Update button to save the settings.

Disable

Check this box to temporarily disable this monitor and any associated alerts. To enable the monitor again, clear the box.

Server

Enter the optional Server name to specify which URLs to check in the URL list. If the URLs are stored in a "map" format, this item is used to check a subset of the URLs from the list. By default, all the URLs in the list are checked.

Log

Enter the path name for the log file for this monitor. For each URL checked, an entry will be added to this log file. If this item is blank, a log is not created.

Error Log

Enter the path name for the error log file for this monitor. For each error retrieving a URL, an entry will be added to this log file. If this item is blank, a log is not created.

Threads

Enter the number of threads to retrieve URLs. This is the number of simultaneous checks to perform. Increasing this number will shorten the time for all of the URLs to be checked but also increase the load on the server.

Pause

Enter the pause, in milliseconds, between each URL check. Decreasing this number will shorten the total time required to check all of the URLs but will also increase the load on the server.

Retries Enter the number of times you want SiteScope to try to reach URLs that are returning an error.

HTTP Proxy

Optionally, a proxy server can be used to access the URLs in the list. Enter the domain name and port of an HTTP Proxy Server.

Authorization User Name

If the URLs in the list require a name and password for access, enter the name in this box.

Authorization Password

If the URLs in the list require a name and password for access, enter the password in this box.

Verify Error

Check this box if you want SiteScope to automatically run this monitor again if it detects an error. When an error is detected, the monitor will immediately be scheduled to run again once.

Note: In order to change the run frequency of this monitor when an error is detected, use the Update every (on errors) option below.

Note: The status returned by the Verify Error run of the monitor will replace the status of the originally scheduled run that detected an error. This may cause the loss of important performance data if the data from the verify run is different than the initial error status.

Warning: Use of this option across many monitor instances may result in significant monitoring delays in the case that multiple monitors are rescheduled to verify errors at the same time.

Update Every (on error)

You use this option to set a new monitoring interval for monitors that have registered an error condition. For example, you may want SiteScope to monitor this item every 10 minutes normally, but as often as every 2 minutes if an error has been detected. Note that this increased scheduling will also affect the number of alerts generated by this monitor.

Schedule

By default, SiteScope monitors are enabled every day of the week. You may, however, schedule your monitors to run only on certain days or on a fixed schedule. Click the Edit schedule link to create or edit a monitor schedule. For more information about working with monitor schedules, see the section on Schedule Preferences for Monitoring.

Monitor Description

Enter additional information about this monitor. The Monitor Description can include HTML tags such as the <BR> <HR>, and <B> tags to control display format and style. The description will appear on the Monitor Detail page.

Report Description

Enter an optional description for this monitor that will make it easier to understand what the monitor does. For example, network traffic or main server response time. This description will be displayed on with each bar chart and graph in Management Reports and appended to the tool-tip displayed when you pass the mouse cursor over the status icon for this monitor on the monitor detail page.

Depends Condition

If you choose to make the running of this monitor dependent on the status of another monitor, select the status condition that the other monitor or monitor group should have in order for the current monitor to run normally. The current monitor will be run normally as long as the monitor on which it depends reports the condition selected in this option.

List Order

By default, new monitors are listed last on the Monitor Detail page. You may use this drop-down list to choose a different placement for this monitor.

Error if

By default, SiteScope generates an error if any of the URLs cannot be successfully retrieved. You may choose to have SiteScope generate an error based on the number of URL errors, the number of URLs checked, or the average retrieval time. Enter a comparison value and use the comparison operator list to specify an error threshold such as: >= (greater than or equal to), != (not equal to), or < (less than).

Warning if

By default, SiteScope does not generate warnings for URL List Monitors. You may choose to have SiteScope generate an error based on the number of URL errors, the number of URLs checked, or the average retrieval time.

Good if

You can use this option to change how SiteScope generates a good status. You can choose to have this based upon duration, number of URLs left to check, or errors.