Skip to main content

Using System Monitor

System Monitor samples important system status and resource usage indicators, such as the status of ECP connections and the percentage of the lock table in use, and generates notifications—alerts, warnings, and “status OK” messages—based on fixed statuses and thresholds. These notifications are written to the messages log, allowing Log Monitor to generate email messages from them if configured to do so. System Monitor also maintains a single overall system health state.

System Monitor is managed using the ^%SYSMONMGR utility.

System Monitor is part of the System Monitor tools.

The System Monitor Process

In each namespace in which it is configured to run, System Monitor gathers and delivers system metric information in three stages using three types of classes (or System Monitor components). Sensor classes collect information, subscriber classes evaluate the information to form notifications, and notifier classes post the notifications to the proper alerting systems. The following describes the sequence in greater depth:

  1. Obtain metric information

    Sensor classes incorporate methods for obtaining the values of system or application metrics. For example, the system sensor class SYS.Monitor.SystemSensors includes the GetProcessCount() method, which returns the number of active processes for the InterSystems IRIS instance, and the GetLockTable() method, which returns the percentage of the instance’s lock table that is in use.

    At a fixed interval, System Monitor calls the GetSensors() method of each configured sensor class. A sensor class may do one of the following:

    • Return an array of sensor name/value pairs to be passed by System Monitor to subscriber classes (described in stage 2)

    • Evaluate the sensor values it obtains and return notifications to be posted by System monitor to notifier classes (described in stage 3)

    One of the sensor classes provided with System Monitor, SYS.Monitor.SystemSensors, returns a name/value array. The other, %SYS.Monitor.AppMonSensor, performs its own evaluations and generates its own notifications.

  2. Evaluate metric information

    Subscriber classes incorporate methods for evaluating sensor values and generating notifications. After calling each sensor class that returns a name/value array, System Monitor calls the Receive() method of each subscriber class, populating the SensorReading property with the array. For each sensor name/value pair provided to its Receive() method, the subscriber class evaluates the value and if appropriate returns a notification containing text and a severity code.

    For example, when System Monitor passes the name/value array returned from SYS.Monitor.SystemSensors.GetSensors() to subscriber classes:

    • the system subscriber, SYS.Monitor.SystemSubscriber, may discover that the LockTablePercentFull value is over 85, its warning threshold for that sensor, and return a notification containing a severity code of 1 and appropriate text.

    • the Health Monitor subscriber, SYS.Monitor.Health.Control, may determine that the ProcessCount value is too high, based on that sensor’s configured parameters and established normal values, and return a notification containing a severity code of 2 and appropriate text.

  3. Generate notifications

    Notifier classes incorporate methods for passing notifications to one or more alerting systems. After calling each sensor class and subscriber class, System Monitor calls the Post() method of each notifier class, populating the Notifications property with the notifications returned by sensor or subscriber classes. The notifier class then passes each notification to the desired alerting method; for example, when the system notifier receives the notifications returned by the system subscriber for LockTablePercentFull and the Health Monitor subscriber for ProcessCount, it writes the severity code and text to the messages log. This approach allows notifications to be passed to independent alerting systems such as the interoperability production alert processors and those in TrakCare, as well as user-defined alerting systems.

System Monitor starts automatically when the instance starts and begins calling the configured sensor classes in each of the configured startup namespaces, passing sensor values to configured subscriber classes and notifications to configured notifier classes in turn. You can define and configure your own System Monitor sensor, subscriber and notifier classes on a per-namespace basis. See the default classes in Default System Monitor Components.

Note:

In an emergency, System Monitor may need to be shut down. The class method %SYS.Monitor.Enabled([flag]) sets, clears, and reports the status of System Monitor. If flag is 0, System Monitor will not start.

Tracking System Monitor Notifications

Typically, any System Monitor alert (notification of severity 2) or sequence of System Monitor warnings (severity 1) should be investigated. Health Monitor can also generate System Monitor alerts and warnings.

System Monitor alerts, warnings, and status messages (severity 0) are written to the messages log (install-dir\mgr\messages.log). (All System Monitor and Health Monitor status messages are written to the System Monitor log, install-dir\mgr\SystemMonitor.log. Application Monitor alerts are not written to logs, but can be sent by email or passed to a specified notification method.)

To track System Monitor alerts and warnings, you can do the following:

  • View System Monitor alerts using the ^%SYSMONMGR utility. This option lets you display alerts for all sensors or for a specific sensor and view all recorded alerts or only those occurring during a specified time period, but it does not display warnings.

  • Monitor the messages log (see Monitoring Log Files). Bear in mind that when a sequence of System Monitor alerts is generated for a given sensor within a short period of time, only the first is written to the messages log.

    Note:

    In the messages log, System Monitor status notifications are labeled with initial capitals, for example [System Monitor] started in %SYS, whereas warnings, alerts and OK messages are labeled in uppercase, such as [SYSTEM MONITOR] CPUusage Warning: CPUusage = 90 ( Warnvalue is 85).

  • Configure Log Monitor to send email notifications of alerts (and optionally warnings) appearing in the messages log (instead of writing them to the alerts log, the default). When relying on this method, keep in mind that Log Monitor does not generate a notification for every messages log entry of the configured severity; when there is a series of entries from a given process (such as System Monitor) within about one hour, a notification is generated for the first entry only. For example, if a network problem causes multiple System Monitor alerts concerning ECP connections and open transactions to be generated over a 15 minute period, Log Monitor generates only one notification (for whichever alert was first). For this reason, on receiving a single System Monitor notification from Log Monitor you should immediately view System Monitor alerts and consult the messages log.

System Monitor Status and Resource Metrics

The following table lists the system status and resource usage metrics sampled by System Monitor, and the notification thresholds and rules for each that result in warnings (severity 1), alerts (severity 2), and “status OK” (severity 0) notifications.

System Monitor Status and Resource Notifications
Metric Description Notification Rules

Disk Space

Available space in a database directory
  • < 250MB — warning

  • < 50MB — alert

  • > 250MB (after warning/alert) — OK

Journal Space

Available space in the journal directory
  • < 250MB — warning

  • < 50MB — alert

  • > 250MB (after warning/alert) — OK

Paging Percentage of physical memory and paging space used
  • paging space > 30% — warning

  • (physical memory > 96%) + (paging space > 50%) — alert

Lock Table Percentage of the lock table in use
  • > 85% — warning

  • > 95% — alert

  • < 85% after warning/alert — OK

write daemon

Status of the write daemon
  • write daemon is awake and processing its (non-empty) queue but has been on one cycle at least 10 seconds longer than the configured write daemon cycle time (default 80 seconds) — alert

  • write daemon completes a pass after alert — OK

ECP Connections

State of connections to ECP application servers or ECP data servers
  • state is Trouble for at least five (5) seconds — alert

Shared Memory Heap (Generic Memory Heap) Status of shared memory heap (SMH), also known as generic memory heap (gmheap)
  • SMH (gmheap) status 1 — warning

  • SMH (gmheap) status 2 — alert

Open Transactions Duration of longest open local or remote (ECP) transactions
  • > 10 minutes — warning

  • > 20 minutes — alert

License Expiration Days until license expires
  • 7 days — warning

  • 5 days or fewer — alert (daily)

SSL/TLS Certificate Expiration Days until certificate expires
  • individual certificate expires within 30 days — warning (repeated daily)

  • one or more daily expiring certificate warnings — alert (summary of warnings, one per day)

ISCAgent (mirror members only) ISCAgent status
  • Unresponsive for <1 minute — warning

  • Unresponsive for >1 minute — alert

System Monitor Health State

Based on notifications posted to the messages log (see Monitoring Log Files), including both system alerts generated directly by the InterSystems IRIS instance and alerts and warnings generated by System Monitor and its Health Monitor component, System Monitor maintains a single value summarizing overall system health in a register in shared memory.

At startup, the system health state is set based on the number of system (not System Monitor) alerts posted to the messages log during the startup process. Once System Monitor is running, the health state can be elevated by either system alerts or System Monitor alerts or warnings. Status is cleared to the next lower level when 30 minutes have elapsed since the last system alert or System Monitor alert or warning was posted. The following table shows how the system health state is determined.

System Monitor Health State
State Set at startup when ... Set following startup when ... Cleared to ...

GREEN (0)

no system alerts are posted during startup 30 minutes (if state was YELLOW) or 60 minutes (if state was RED) have elapsed since the last system alert or System Monitor alert or warning was posted n/a

YELLOW (1)

up to four system alerts are posted during startup state is GREEN and
  • one system alert is posted

    OR

  • one or more System Monitor alerts and/or warnings are posted, but not alerts sufficient to set RED, as below

GREEN when 30 minutes have elapsed since the last system alert or System Monitor alert or warning was posted

RED (2)

five or more system alerts are posted during startup
  • state is YELLOW and one system alert is posted

    OR

  • state is GREEN or YELLOW and during a 30 minute period, System Monitor alerts from at least five different sensors or three System Monitor alerts from a single sensor are posted

YELLOW when 30 minutes have elapsed since the last system alert or System Monitor alert or warning was posted
Note:

A fourth state, HUNG, can occur when global updates are blocked. Specifically, the following events change the state to HUNG:

  • The journal daemon is paused for more than 5 seconds or frozen (see Journal I/O Errors).

  • Any of switches 10, 11, 13, or 14 are set (see Using Switches).

  • The write daemon is stopped for any reason or sets the updates locked flag for more than 3 seconds.

  • The number of available global buffers (in the database cache) falls into the critical region and remains there for more than 5 seconds.

When the health state changes to HUNG, the reason is written to the messages log.

You can view the System Monitor health state using:

  • The View System Health option on the View System Data menu of ^%SYSMONMGR (which does not report HUNG).

  • The $SYSTEM.Monitor API, which lets you access the system status directly. Use $SYSTEM.Monitor.State()Opens in a new tab to return the system status; see also the SetState, Clear, Alert, GetAlerts, and ClearAlerts methods.

  • The iris list and iris qlist commands (which do not include health state on Windows).

Note:

When System Monitor is not running, the System Monitor health state is always GREEN.

System Monitor Defaults

System Monitor calls a provided set of classes that can be augmented, runs in the %SYS namespace, and operates under three default settings that can be changed.

Default System Monitor Components

Five classes are provided with InterSystems IRIS and configured in System Monitor in the %SYS namespace by default.

Sensor classes:

  • SYS.Monitor.SystemSensors

    System sensor class obtaining sensor values to be passed to configured subscriber classes, including the System Monitor subscriber (SYS.Monitor.SystemSubscriber) and Health Monitor subscriber (SYS.Monitor.Health.Control).

  • %SYS.Monitor.AppMonSensor

    Class providing sensor, subscriber and notification services for Application Monitor; obtains sensor values and stores them in the local namespace, evaluates the values based on user-defined alerts and either generates an email message or calls a user-specified method when an alert is triggered, based on the alert definition.

Subscriber classes:

  • SYS.Monitor.Health.Control

    Subscriber class for Health Monitor; receives and evaluates statistical sensor values from SYS.Monitor.SystemSensors and posts notifications to the system notifier.

  • SYS.Monitor.SystemSubscriber

    System Monitor subscriber available to all sensor classes; contains all code required to monitor and analyze the sensors in SYS.Monitor.SystemSensors. Generates System Monitor notifications and Health Monitor notifications for some sensors.

Notifier class:

  • SYS.Monitor.SystemNotifyOpens in a new tab

    System notifier available to all subscriber classes. On receiving a notification from the system subscriber (SYS.Monitor.SystemSubscriber) or Health Monitor subscriber (SYS.Monitor.Health.Control), writes it to the System Monitor log, and to the messages log if it is of severity 2 (alert). (See Monitoring InterSystems IRIS Using the Management Portal for information on these log files.)

    The system notifier also generates a single overall evaluation of system status that can be obtained using the SYS.Monitor.State() method, which returns 0 (GREEN), 1 (YELLOW), or 2 (RED).

User-defined classes can be configured using ^%SYSMONMGR.

Default System Monitor Namespace

All System Monitor and Application Monitor configurations and settings are namespace-specific. By default, System Monitor starts and runs only in the %SYS namespace. Additional startup namespaces for System Monitor and Application Monitor can be configured using ^%SYSMONMGR. Following any change you make to the System Monitor or Application Monitor configuration for a namespace, you must restart System Monitor in the namespace for the change to take effect.

Health Monitor runs only in the %SYS namespace.

Default System Monitor Settings

By default, the System Monitor is always running when the instance is running; it can be stopped using ^%SYSMONMGR but will start automatically again when the instance next starts.

By default, the System Monitor:

  • calls the GetSensors() method of each configured sensor class every 30 seconds.

  • writes only alerts, warnings and messages to the System Monitor log, and does not write sensor readings.

  • does not save sensor readings.

These settings can be changed using ^%SYSMONMGR.

Using the ^%SYSMONMGR Utility

The ^%SYSMONMGR utility lets you manage and configure the System Monitor. The utility can be executed in any namespace, and changes made with it affect only the namespace in which it is started. You must maintain a separate System Monitor configuration for each startup namespace you configure by executing ^%SYSMONMGR in that namespace. Following any change you make to the System Monitor configuration for a namespace, you must restart System Monitor in the namespace for the change to take effect.

Important:

All manual operations using the ^%SYSMONMGR utility described in this section can be executed programatically using the methods in the %Monitor.Manager API.

To manage the System Monitor, enter the following command in the Terminal:

do ^%SYSMONMGR

The main menu appears.

1) Start/Stop System Monitor
2) Set System Monitor Options
3) Configure System Monitor Classes
4) View System Monitor State
5) Manage Application Monitor
6) Manage Health Monitor
7) View System Data
8) Exit 

Option? 

Enter the number of your choice or press Enter to exit the utility.

The options in the main menu let you perform System Monitor tasks as described in the following table:

Option Description
1) Start/Stop System Monitor
  • Start System Monitor

  • Stop System Monitor

2) Set System Monitor Options
  • Set the sampling interval for configured sensor classes

  • Set the debugging level of information written to the System Monitor log

  • Enable saving of sensor readings and set number of days to save

  • Return sampling interval, debugging level, and sensor reading saving to their defaults

3) Configure System Monitor Components
  • Configure or remove user-defined sensor, subscriber and notifier classes

  • Configure startup namespaces

4) View System Monitor State
  • Display the operating status of System Monitor and its configured components

5) Manage Application Monitor
  • Display the Application Monitor submenu

6) Manage Health Monitor
  • Display the Health Monitor submenu (available only if ^%SYSMONMGR is run in the %SYS namespace)

7) View System Data

Start/Stop System Monitor

When an InterSystems IRIS instance starts, System Monitor starts automatically and begins calling configured classes in each configured startup namespace; this cannot be changed. While the instance is running, however, you can stop System Monitor, and must do so in order to change the configuration of Health Monitor. In addition, following any change you make to the System Monitor configuration for a namespace, you must restart System Monitor in the namespace for the change to take effect.

When you enter 1 at the main menu, the following menu is displayed:

1) Start System Monitor
2) Stop System Monitor
3) Exit

Enter 2 to stop System Monitor when it is running, and 1 to start it when it is stopped.

Note:

System Monitor monitors the size of the messages log and rolls it over when required. When System Monitor is stopped, the messages log may exceed the limit set by the MaxConsoleLogSize configuration setting until the instance is restarted or the PurgeErrorsAndLogs task is run. See Monitoring Log Filesfor information about the messages log.

Set System Monitor Options

To change global System Monitor settings or to return them to their default values, stop System Monitor if it is running and then enter 2 at the main menu:

1) Set Sample Interval
2) Set Debugging Level
3) Reset Defaults
4) Manage Debug Data
5) Exit

Enter 1 to set the interval at which System Monitor calls each configured sensor class; the default is 30 seconds.

Enter 2 to set the debugging level. The default is 0 (base) which writes System Monitor and Health monitor status and error messages to the System Monitor log, and does not save sensor readings. Debugging level 1 (log all sensors) writes sensor readings to the System Monitor log along with messages and saves sensor readings, which can then be viewed using the View Sensor Data option of the View System Data menu.

Enter 3 to reset the sample interval, debugging level, and saving of sensor readings to their default values.

Enter 4 to set the number of days for which sensor readings are saved; the default is 5.

Your changes take effect when you next start or restart System Monitor.

Configure System Monitor Components

As described in System Monitor, you can create your own sensor, subscriber and notifier classes by extending %SYS.Monitor.AbstractSensorOpens in a new tab, %SYS.Monitor.AbstractSubscriberOpens in a new tab, and %SYS.Monitor.AbstractNotificationOpens in a new tab, respectively, and configure them in System Monitor to extend the capabilities of the provided classes described in Default System Monitor Components. You can also add namespaces other than %SYS for System Monitor to start and run in.

Configure System Monitor Classes

When you enter 3 at the main menu, the following menu is displayed:

1) Configure Components
2) Configure Startup Namespaces
3) Exit

Enter 1 to display the options for configuring classes:

1) List Classes
2) Add Class
3) Delete Class
4) Exit

Enter 1 to list the currently configured classes for the namespace in which you started ^%SYSMONMGR, including provided system classes and those you have configured.

Enter 2 to configure a user-defined class for the namespace in which you started ^%SYSMONMGR. The class you specify must exist in the namespace and be recognized by System Monitor as a valid sensor, subscriber or notifier class. You can also enter a description of the class.

Enter 3 to delete a user-defined class you have configured.

Note:

Configuring or deleting a class affects only the namespace in which you started ^%SYSMONMGR.

Configure System Monitor Namespaces

When an instance starts, System Monitor automatically starts as a separate process in each configured startup namespace (by default %SYS only). All System Monitor configurations and settings are namespace-specific. When you make changes using ^%SYSMONMGR, the changes affect only the namespace in which you started the utility.

Note:

All instances of ^%SYSMONMGR write messages to the same System Monitor log. Startup namespaces can be configured from any namespace.

When you enter 3 at the main menu, the following menu is displayed:

1) Configure Components
2) Configure Startup Namespaces
3) Exit

Enter 2 to display the options for configuring namespaces:

1) List Startup Namespaces
2) Add Namespace
3) Delete Namespace
4) Exit

Enter 1 to list the currently configured startup namespaces.

Enter 2 to add a startup namespace.

Enter 3 to delete a startup namespace. (You cannot delete %SYS.)

View System Monitor State

Enter 4 at the main menu to display the status of System Monitor and its components in the namespace in which you started ^%SYSMONMGR, for example:

       Component                   State
System Monitor                     OK
%SYS.Monitor.AppMonSensor          None
SYS.Monitor.SystemSensors          OK
SYS.Monitor.Health.Control         Running: Period is Thursday 09:00 - 11:30
SYS.Monitor.SystemSubscriber       OK
SYS.Monitor.SystemNotifier         OK

In this example, System Monitor and its system sensor, subscriber and notifier classes are running normally, as is Health Monitor’s subscriber class. However, none of Application Monitor’s classes are activated (see Manage Monitor Classes), so it is not evaluating sensor samples or generating alerts.

Manage Application Monitor

Manage Health Monitor

View System Data

Enter 7 at the main menu (or 6 in namespaces other than %SYS) to display options for viewing System Monitor information about the system.

1) View Sensor Data
2) View System Health
3) View Alerts
4) Exit

Enter 1 to view saved sensor readings, if you have enabled saving of sensor data using the Manage Debug Data option on the Set System Monitor Options menu. You can display saved readings for all sensors or for a specific sensor, and you can view all saved sensor readings or only those for a time period you specify.

Enter 2 to view the System Monitor health state, including all alerts between the previous GREEN state and the current state, if not GREEN.

Enter 3 to view System Monitor alerts. You can display alerts for all sensors or for a specific sensor, and you can view all alerts within the period you specified using the Manage Debug Data option on the Set System Monitor Options menu, or only those for a time period you specify.

Defining System Monitor Components

The SYS.Monitor API lets define your own sensor, subscriber, and notifier classes.

Sensor Classes

Sensor classes extend %SYS.Monitor.AbstractSensorOpens in a new tab. The System Monitor controller initially calls each sensor class’s Start() method; thereafter, on each cycle, it calls the GetSensors() method. The SetSensor() method is used within the sensor class to set sensor name/value pairs in the SensorReading property, which is returned by GetSensors() and passed to all subscriber classes.

A sensor class may also evaluate sensor readings and, as a result of its evaluation, call the %SYS.Monitor.EmailOpens in a new tab class for generating email messages from notifications or any user-defined alerting method.

Subscriber Classes

Subscriber classes extend %SYS.Monitor.AbstractSubscriberOpens in a new tab. The System Monitor controller initially calls each subscriber class’s Start() method; thereafter, on each cycle, it calls the Receive() method once for each sensor class called in the cycle, passing the SensorReading property with the sensor name/value pairs received from that sensor class. The subscriber class may evaluate one or more of the name/value pairs and set notifications using the Notify() method, which populates the Notifications property.

A subscriber class may also, as a result of its sensor evaluation, call the %SYS.Monitor.EmailOpens in a new tab class for generating email messages from notifications, or any user-defined alerting method.

%SYS.Monitor.SampleSubscriberOpens in a new tab is provided as a sample subscriber class.

Notifier Classes

Notifier classes extend %SYS.Monitor.AbstractNotificationOpens in a new tab. The System Monitor controller initially calls each notifier class’s Start() method; thereafter, on each cycle, it calls the Post() method once for each subscriber class called in the cycle, passing the Notifications property with the notifications received from that subscriber. The notifier class calls then passes the notifications to its alerting method(s), which may include the %SYS.Monitor.EmailOpens in a new tab class for generating email messages from notifications or any user-defined alerting method.

See Also

FeedbackOpens in a new tab