Using System Monitor

The System Monitor Process

In each namespace in which it is configured to run, System Monitor gathers and delivers system metric information in three stages using three types of classes (or System Monitor components). Sensor classes collect information, subscriber classes evaluate the information to form notifications, and notifier classes post the notifications to the proper alerting systems. The following describes the sequence in greater depth:

1. Obtain metric information

   Sensor classes incorporate methods for obtaining the values of system or application metrics. For example, the system sensor class SYS.Monitor.SystemSensors includes the GetProcessCount() method, which returns the number of active processes for the InterSystems IRIS instance, and the GetLockTable() method, which returns the percentage of the instance’s lock table that is in use.

   At a fixed interval, System Monitor calls the GetSensors() method of each configured sensor class. A sensor class may do one of the following:

   • Return an array of sensor name/value pairs to be passed by System Monitor to subscriber classes (described in stage 2)

   • Evaluate the sensor values it obtains and return notifications to be posted by System monitor to notifier classes (described in stage 3)

   One of the sensor classes provided with System Monitor, SYS.Monitor.SystemSensors, returns a name/value array. The other, %SYS.Monitor.AppMonSensor, performs its own evaluations and generates its own notifications.

2. Evaluate metric information

   Subscriber classes incorporate methods for evaluating sensor values and generating notifications. After calling each sensor class that returns a name/value array, System Monitor calls the Receive() method of each subscriber class, populating the SensorReading property with the array. For each sensor name/value pair provided to its Receive() method, the subscriber class evaluates the value and if appropriate returns a notification containing text and a severity code.

   For example, when System Monitor passes the name/value array returned from SYS.Monitor.SystemSensors.GetSensors() to subscriber classes:

   • the system subscriber, SYS.Monitor.SystemSubscriber, may discover that the LockTablePercentFull value is over 85, its warning threshold for that sensor, and return a notification containing a severity code of 1 and appropriate text.

   • the Health Monitor subscriber, SYS.Monitor.Health.Control, may determine that the ProcessCount value is too high, based on that sensor’s configured parameters and established normal values, and return a notification containing a severity code of 2 and appropriate text.

3. Generate notifications

   Notifier classes incorporate methods for passing notifications to one or more alerting systems. After calling each sensor class and subscriber class, System Monitor calls the Post() method of each notifier class, populating the Notifications property with the notifications returned by sensor or subscriber classes. The notifier class then passes each notification to the desired alerting method; for example, when the system notifier receives the notifications returned by the system subscriber for LockTablePercentFull and the Health Monitor subscriber for ProcessCount, it writes the severity code and text to the messages log. This approach allows notifications to be passed to independent alerting systems such as the interoperability production alert processors and those in TrakCare, as well as user-defined alerting systems.

System Monitor starts automatically when the instance starts and begins calling the configured sensor classes in each of the configured startup namespaces, passing sensor values to configured subscriber classes and notifications to configured notifier classes in turn. You can define and configure your own System Monitor sensor, subscriber and notifier classes on a per-namespace basis. See the default classes in Default System Monitor Components.

Tracking System Monitor Notifications

Typically, any System Monitor alert (notification of severity 2) or sequence of System Monitor warnings (severity 1) should be investigated. Health Monitor can also generate System Monitor alerts and warnings.

System Monitor alerts, warnings, and status messages (severity 0) are written to the messages log (install-dir\mgr\messages.log). (All System Monitor and Health Monitor status messages are written to the System Monitor log, install-dir\mgr\SystemMonitor.log. Application Monitor alerts are not written to logs, but can be sent by email or passed to a specified notification method.)

To track System Monitor alerts and warnings, you can do the following:

• View System Monitor alerts using the ^%SYSMONMGR utility. This option lets you display alerts for all sensors or for a specific sensor and view all recorded alerts or only those occurring during a specified time period, but it does not display warnings.

• Monitor the messages log (see Monitoring Log Files). Bear in mind that when a sequence of System Monitor alerts is generated for a given sensor within a short period of time, only the first is written to the messages log.

  Note:

  In the messages log, System Monitor status notifications are labeled with initial capitals, for example [System Monitor] started in %SYS, whereas warnings, alerts and OK messages are labeled in uppercase, such as [SYSTEM MONITOR] CPUusage Warning: CPUusage = 90 ( Warnvalue is 85).

• Configure Log Monitor to send email notifications of alerts (and optionally warnings) appearing in the messages log (instead of writing them to the alerts log, the default). When relying on this method, keep in mind that Log Monitor does not generate a notification for every messages log entry of the configured severity; when there is a series of entries from a given process (such as System Monitor) within about one hour, a notification is generated for the first entry only. For example, if a network problem causes multiple System Monitor alerts concerning ECP connections and open transactions to be generated over a 15 minute period, Log Monitor generates only one notification (for whichever alert was first). For this reason, on receiving a single System Monitor notification from Log Monitor you should immediately view System Monitor alerts and consult the messages log.

System Monitor Status and Resource Metrics

The following table lists the system status and resource usage metrics sampled by System Monitor, and the notification thresholds and rules for each that result in warnings (severity 1), alerts (severity 2), and “status OK” (severity 0) notifications.

System Monitor Health State

Based on notifications posted to the messages log (see Monitoring Log Files), including both system alerts generated directly by the InterSystems IRIS instance and alerts and warnings generated by System Monitor and its Health Monitor component, System Monitor maintains a single value summarizing overall system health in a register in shared memory.

At startup, the system health state is set based on the number of system (not System Monitor) alerts posted to the messages log during the startup process. Once System Monitor is running, the health state can be elevated by either system alerts or System Monitor alerts or warnings. Status is cleared to the next lower level when 30 minutes have elapsed since the last system alert or System Monitor alert or warning was posted. The following table shows how the system health state is determined.

You can view the System Monitor health state using:

• The View System Health option on the View System Data menu of ^%SYSMONMGR (which does not report HUNG).

• The $SYSTEM.Monitor API, which lets you access the system status directly. Use $SYSTEM.Monitor.State()Opens in a new tab to return the system status; see also the SetState, Clear, Alert, GetAlerts, and ClearAlerts methods.

• The iris list and iris qlist commands (which do not include health state on Windows).

System Monitor Defaults

System Monitor calls a provided set of classes that can be augmented, runs in the %SYS namespace, and operates under three default settings that can be changed.

Using the ^%SYSMONMGR Utility

The ^%SYSMONMGR utility lets you manage and configure the System Monitor. The utility can be executed in any namespace, and changes made with it affect only the namespace in which it is started. You must maintain a separate System Monitor configuration for each startup namespace you configure by executing ^%SYSMONMGR in that namespace. Following any change you make to the System Monitor configuration for a namespace, you must restart System Monitor in the namespace for the change to take effect.

To manage the System Monitor, enter the following command in the Terminal:

do ^%SYSMONMGR

The main menu appears.

1) Start/Stop System Monitor
2) Set System Monitor Options
3) Configure System Monitor Classes
4) View System Monitor State
5) Manage Application Monitor
6) Manage Health Monitor
7) View System Data
8) Exit 

Option? 

Enter the number of your choice or press Enter to exit the utility.

The options in the main menu let you perform System Monitor tasks as described in the following table:

Start/Stop System Monitor

When an InterSystems IRIS instance starts, System Monitor starts automatically and begins calling configured classes in each configured startup namespace; this cannot be changed. While the instance is running, however, you can stop System Monitor, and must do so in order to change the configuration of Health Monitor. In addition, following any change you make to the System Monitor configuration for a namespace, you must restart System Monitor in the namespace for the change to take effect.

When you enter 1 at the main menu, the following menu is displayed:

1) Start System Monitor
2) Stop System Monitor
3) Exit

Enter 2 to stop System Monitor when it is running, and 1 to start it when it is stopped.

Note:

System Monitor monitors the size of the messages log and rolls it over when required. When System Monitor is stopped, the messages log may exceed the limit set by the MaxConsoleLogSize configuration setting until the instance is restarted or the PurgeErrorsAndLogs task is run. See Monitoring Log Filesfor information about the messages log.

Set System Monitor Options

To change global System Monitor settings or to return them to their default values, stop System Monitor if it is running and then enter 2 at the main menu:

1) Set Sample Interval
2) Set Debugging Level
3) Reset Defaults
4) Manage Debug Data
5) Exit

Enter 1 to set the interval at which System Monitor calls each configured sensor class; the default is 30 seconds.

Enter 2 to set the debugging level. The default is 0 (base) which writes System Monitor and Health monitor status and error messages to the System Monitor log, and does not save sensor readings. Debugging level 1 (log all sensors) writes sensor readings to the System Monitor log along with messages and saves sensor readings, which can then be viewed using the View Sensor Data option of the View System Data menu.

Enter 3 to reset the sample interval, debugging level, and saving of sensor readings to their default values.

Enter 4 to set the number of days for which sensor readings are saved; the default is 5.

Your changes take effect when you next start or restart System Monitor.

Configure System Monitor Components

As described in System Monitor, you can create your own sensor, subscriber and notifier classes by extending %SYS.Monitor.AbstractSensorOpens in a new tab, %SYS.Monitor.AbstractSubscriberOpens in a new tab, and %SYS.Monitor.AbstractNotificationOpens in a new tab, respectively, and configure them in System Monitor to extend the capabilities of the provided classes described in Default System Monitor Components. You can also add namespaces other than %SYS for System Monitor to start and run in.

Configure System Monitor Classes

When you enter 3 at the main menu, the following menu is displayed:

1) Configure Components
2) Configure Startup Namespaces
3) Exit

Enter 1 to display the options for configuring classes:

1) List Classes
2) Add Class
3) Delete Class
4) Exit

Enter 1 to list the currently configured classes for the namespace in which you started ^%SYSMONMGR, including provided system classes and those you have configured.

Enter 2 to configure a user-defined class for the namespace in which you started ^%SYSMONMGR. The class you specify must exist in the namespace and be recognized by System Monitor as a valid sensor, subscriber or notifier class. You can also enter a description of the class.

Enter 3 to delete a user-defined class you have configured.

Note:

Configuring or deleting a class affects only the namespace in which you started ^%SYSMONMGR.

Configure System Monitor Namespaces

When an instance starts, System Monitor automatically starts as a separate process in each configured startup namespace (by default %SYS only). All System Monitor configurations and settings are namespace-specific. When you make changes using ^%SYSMONMGR, the changes affect only the namespace in which you started the utility.

Note:

All instances of ^%SYSMONMGR write messages to the same System Monitor log. Startup namespaces can be configured from any namespace.

When you enter 3 at the main menu, the following menu is displayed:

1) Configure Components
2) Configure Startup Namespaces
3) Exit

Enter 2 to display the options for configuring namespaces:

1) List Startup Namespaces
2) Add Namespace
3) Delete Namespace
4) Exit

Enter 1 to list the currently configured startup namespaces.

Enter 2 to add a startup namespace.

Enter 3 to delete a startup namespace. (You cannot delete %SYS.)

View System Monitor State

Enter 4 at the main menu to display the status of System Monitor and its components in the namespace in which you started ^%SYSMONMGR, for example:

       Component                   State
System Monitor                     OK
%SYS.Monitor.AppMonSensor          None
SYS.Monitor.SystemSensors          OK
SYS.Monitor.Health.Control         Running: Period is Thursday 09:00 - 11:30
SYS.Monitor.SystemSubscriber       OK
SYS.Monitor.SystemNotifier         OK

In this example, System Monitor and its system sensor, subscriber and notifier classes are running normally, as is Health Monitor’s subscriber class. However, none of Application Monitor’s classes are activated (see Manage Monitor Classes), so it is not evaluating sensor samples or generating alerts.

View System Data

Enter 7 at the main menu (or 6 in namespaces other than %SYS) to display options for viewing System Monitor information about the system.

1) View Sensor Data
2) View System Health
3) View Alerts
4) Exit

Enter 1 to view saved sensor readings, if you have enabled saving of sensor data using the Manage Debug Data option on the Set System Monitor Options menu. You can display saved readings for all sensors or for a specific sensor, and you can view all saved sensor readings or only those for a time period you specify.

Enter 2 to view the System Monitor health state, including all alerts between the previous GREEN state and the current state, if not GREEN.

Enter 3 to view System Monitor alerts. You can display alerts for all sensors or for a specific sensor, and you can view all alerts within the period you specified using the Manage Debug Data option on the Set System Monitor Options menu, or only those for a time period you specify.

Defining System Monitor Components

The SYS.Monitor API lets define your own sensor, subscriber, and notifier classes.

See Also

• Introduction to System Monitor

• Using Core System Monitor

• Using Health Monitor

• Application Monitor

• System Monitoring Tools

• Manage Email Options (information about generating email messages from notifications in the messages log, including those generated by System Monitor)

• Monitoring Log Files (includes information on the log files generated by this tool)