ICM Node Types

Role DATA: Sharded Cluster Data Node

As described in Overview of InterSystems IRIS ShardingOpens in a new tab and other sections of the “Horizontally Scaling for Data Volume with Sharding” chapter of the Scalability Guide, a typical sharded cluster consists only of data nodes, across which the sharded data is partitioned, and therefore requires only a DATA node definition in the definitions.json file. If DATA nodes are defined, the deployment must be a sharded cluster, and the only other node type that can be defined with them is COMPUTE.

DATA nodes can be mirrored if provisioned in a number matching the MirrorMap setting in their definition, as described in Rules for Mirroring. The DATA nodes in a cluster must be either all mirrored or all nonmirrored.

The only distinction between data nodes in a sharded cluster is that the first node configured (known as node 1) stores all of the nonsharded data, metadata, and code for the cluster in addition to its share of the sharded data. The difference in storage requirements, however, is typically very small. Because all data, metadata, and code is visible on any node in the cluster, application connections can be load balanced across all of the nodes to take greatest advantage of parallel query processing and partitioned caching. A load balancer may be assigned to DATA nodes; see Role LB: Load Balancer.

Role COMPUTE: Sharded Cluster Compute Node

For advanced use cases in which extremely low query latencies are required, potentially at odds with a constant influx of data, compute nodes can be added to a sharded cluster to provide a transparent caching layer for servicing queries, separating the query and data ingestion workloads and improving the performance of both. (For more information see Deploy Compute Nodes for Workload Separation and Increased Query ThroughputOpens in a new tab in the Scalability Guide.)

Adding compute nodes yields significant performance improvement only when there is at least one compute node per data node, so you should define at least as many COMPUTE nodes as DATA nodes; if the number of DATA nodes in the definitions file is greater than the number of COMPUTE nodes, ICM issues a warning. Configuring multiple compute nodes per data node can further improve the cluster’s query throughput, and the recommended best practice when doing so is to configure the same number of compute nodes for each data node, so ICM distributes the defined COMPUTE nodes as evenly as possible across the DATA nodes.

Because COMPUTE nodes support query execution only and do not store any data, their instance type and other settings can be tailored to suit those needs, for example by emphasizing memory and CPU and keeping storage to the bare minimum. Because they do not store data, COMPUTE nodes cannot be mirrored.

A load balancer may be assigned to COMPUTE nodes; see Role LB: Load Balancer.

Role DM: Distributed Cache Cluster Data Server, Standalone Instance, Shard Master Data Server

If multiple nodes of role AM and a DM node (nonmirrored or mirrored) are specified, they are deployed as an InterSystems IRIS distributed cache cluster, with the former serving as application servers and the latter as an data server.

A node of role DM (nonmirrored or mirrored) deployed by itself becomes a standalone InterSystems IRIS instance.

If a DM node (mirrored or nonmirrored), DS nodes (mirrored or nonmirrored), and (optionally) QS nodes are specified, they are deployed as a namespace-levelOpens in a new tab sharded cluster.

Role DS: Shard Data Server

Under the namespace-levelOpens in a new tab architecture, a data shard stores one horizontal partition of each sharded table loaded into a sharded cluster. A node hosting a data shard is called a shard data server. A cluster can have two or more shard data servers up to over 200. Shard data servers can be mirrored by deploying an even number and specifying mirroring.

Role QS: Shard Query Server

Under the namespace-level architecture, shard query servers provides query access to the data shards to which they are assigned, minimizing interference between query and data ingestion workloads and increasing the bandwidth of a sharded cluster for high volume multiuser query workloads. If shard query servers are deployed they are assigned round-robin to the deployed shard data servers. Shard query servers automatically redirect application connections when a mirrored shard data server fails over.

If QS nodes are defined but DS nodes are not, ICM responds with an arror like the following:

Shard Query Server (role 'QS') requires at least one Shard Data Server (role 'DS')

Role AM: Distributed Cache Cluster Application Server

If multiple nodes of role AM and a DM node are specified, they are deployed as an InterSystems IRIS distributed cache cluster, with the former serving as application servers and the latter as a data server. When the data server is mirrored, application connection redirection following failover is automatic.

A load balancer may be assigned to AM nodes; see Role LB: Load Balancer.

Role AR: Mirror Arbiter

When DATA nodes (sharded cluster DATA nodes), a DM node (distributed cache cluster data server, stand-alone InterSystems IRIS instance, or namespace-level shard master data server), or DS nodes (namespace-level shard data servers) are mirrored, deployment of an arbiter node to facilitate automatic failover is highly recommended. One arbiter node is sufficient for all of the mirrors in a cluster; multiple arbiters are not supported and are ignored by ICM, as are arbiter nodes in a nonmirrored cluster.

The AR node does not contain an InterSystems IRIS instance, using a different image to run an ISCAgent container. This arbiter image must be specified using the DockerImage field in the definitions file entry for the AR node; for more information, see The icm run Command.

For more information about the arbiter, see the “MirroringOpens in a new tab” chapter of the High Availability Guide.

Role WS: Web Server

A deployment may contain any number of web servers. Each web server node contains an InterSystems Web Gateway installation along with an Apache web server. ICM populates the remote server list in the InterSystems Web Gateway as follows:

• If DATA and COMPUTE nodes are deployed (node-level sharded cluster), all of the DATA and COMPUTE nodes, or all of the DATA nodes if no COMPUTE nodes are deployed.

• If AM nodes are deployed (distributed cache cluster), all of the AM nodes.

• Otherwise, the DM node (standalone instance or namespace-level sharded cluster).

  Note:

  If deploying a namespace-level sharded cluster with a web server tier, you can manually deploy a custom or third-party load balancer to distribute connections across the DS and (if they exist) QS nodes of the cluster (as recommended in Deploying the Namespace-level ArchitectureOpens in a new tab in the Scalability Guide) and manually edit the Web Gateway configurations to populate the remote server lists with the load balancer address (for more information, see Mirrored Configurations, Failover, and Load BalancingOpens in a new tab in the Web Gateway Guide).

For mirrored DATA and DM nodes, a mirror-aware connection is created, and application connection redirection following failover is automatic. Communication between the web server and the remote servers is configured to run in TLS mode.

A load balancer may be assigned to WS nodes; see Role LB: Load Balancer.

The WS node does not contain an InterSystems IRIS instance, using a different image to run a Web Gateway container. As described in The icm run Command, the webgateway image can be specified by including the DockerImage field in the WS node definition in the definitions.json file, for example:

{
    "Role": "WS",
    "Count": "3",
    "DockerImage": "intersystems/webgateway:latest-em",
    "ApplicationPath": "/acme",
    "AlternativeServers": "LoadBalancing"
}

If the ApplicationPath field is provided, its value is used to create an application path for each instance of the Web Gateway. The default server for this application path is assigned round-robin across Web Gateway instances, with the remaining remote servers making up the alternative server pool. For example, if the preceding sample WS node definition were part of a distributed cache cluster with three AM nodes, the assignments would be like the following:

The AlternativeServers field determines how the Web Gateway distributes requests to its target server pool. Valid values are LoadBalancing (the default) and FailOver. This field has no effect if the ApplicationPath field is not specified.

For information about using the InterSystems Web Gateway, see the Web Gateway GuideOpens in a new tab.

Role SAM: System Alerting and Monitoring Node

Defining a SAM node adds the System Alerting and Monitoring (SAM) cluster monitoring solution to a deployment. For information about adding SAM, see Monitoring in ICM; for complete information about SAM, see the System Alerting and Monitoring GuideOpens in a new tab.

Role LB: Load Balancer

ICM automatically provisions a predefined load balancer node when the provisioning platform is AWS, GCP, Azure, or Tencent, and the definition of nodes of type DATA, COMPUTE, DM, AM, or WS in the definitions file sets LoadBalancer to true. For a generic load balancer for VM or CN nodes, additional parameters must be provided.

Predefined Load Balancer

For nodes of role LB, ICM configures the ports and protocols to be forwarded as well as the corresponding health checks. Queries can be executed against the deployed load balancer the same way one would against a data node in a sharded cluster or a distributed cache cluster application server.

To add a load balancer to the definition of DATA, COMPUTE, DM, AM, or WS nodes, add the LoadBalancer field, for example:

{
    "Role": "AM",
    "Count": "2",
    "LoadBalancer": "true"
}

The following example illustrates the nodes that would be created and deployed given this definition:

$ icm inventory
Machine           IP Address    DNS Name                              Region   Zone
-------           ----------    --------                              ------   ----
ANDY-AM-TEST-0001 54.214.230.24 ec2-54-214-230-24.amazonaws.com       us-west1 c
ANDY-AM-TEST-0002 54.214.230.25 ec2-54-214-230-25.amazonaws.com       us-west1 c
ANDY-LB-TEST-0000 (virtual AM)  ANDY-AM-TEST-1546467861.amazonaws.com us-west1 c

Queries against this cluster can be executed against the load balancer the same way they would be against the AM nodes.

Predefined load balancers for mirrored DATA and DM nodes are mirror-aware and always direct traffic to the current primary.

The LoadBalancer field can be added to more than one definition in a deployment; for example a distributed cache cluster can contain AM nodes receiving load-balanced connections from a WS tier that receives load-balanced application connections.

Currently, a single automatically provisioned load balancer cannot serve multiple node types (for example, both DATA and COMPUTE nodes), so each requires its own load balancer. This does not preclude the user from manually deploying a custom or third-party load balancer to serve the desired roles. Another useful approach is to provision a load balancer for WS nodes, which can then distribute application connections across multiple node types as described in Role WS: Web Server.

Note:

When provisioning on AWS, you can specify a load balancer of type “internal” by setting LoadBalancerInternal to True in the definition in which LoadBalancer is set to True.

Generic Load Balancer

A load balancer can be added to VM (virtual machine) and CN (container) nodes by providing the following additional keys:

• ForwardProtocol

• ForwardPort

• HealthCheckProtocol

• HealthCheckPath

• HealthCheckPort

The following is an example:

{
    "Role": "VM",
    "Count": "2",
    "LoadBalancer": "true",
    "ForwardProtocol": "tcp",
    "ForwardPort": "443",
    "HealthCheckProtocol": "http",
    "HealthCheckPath": "/csp/status.cxw",
    "HealthCheckPort": "8080"
}

ForwardPort can be a comma-separated list of ports to forward, with the condition that all of the forwarded ports share the same ForwardProtocol.

More information about these keys can be found in the Ports and Protocol Parameters table in “ICM Configuration Parameters”.

Role VM: Virtual Machine Node

A cluster may contain any number of virtual machine nodes. A virtual machine node provides a means of allocating host nodes which do not have a predefined role within an InterSystems IRIS cluster. Docker is not installed on these nodes, though users are free to deploy whatever custom or third-party software (including Docker) they wish.

The following commands are supported on the virtual machine node:

• icm provision

• icm unprovision

• icm inventory

• icm ssh

• icm scp

A load balancer may be assigned to VM nodes; see Role LB: Load Balancer.

Role CN: Container Node

A cluster may contain any number of container nodes. A container node is a general purpose node with Docker installed.

You can add InterSystems API Manager (IAM) to any deployment by defining a CN node and including the IAM and IAMImage fields; for more information, see Deploying InterSystems API Manager (IAM) in the “ICM Reference” chapter. You can also deploy any custom and third-party containers you wish on a CN node; iris (InterSystems IRIS) containers will not be deployed if specified. All ICM commands are supported for container nodes, but most will be filtered out unless they use the -container option to specify a container other than iris, or either the -role or -machine option to limit the command to CN nodes (see ICM Commands and Options).

A load balancer may be assigned to CN nodes; see Role LB: Load Balancer. CN nodes cannot be deployed in containerless mode.

Role BH: Bastion Host

You may want to deploy a configuration that offers no public network access. If you have an existing private network, you can launch ICM on a node on that network and deploy within it. If you do not have such a network, you can have ICM configure a private subnet and deploy your configuration on it. Since ICM is not running within that private subnet, however, it needs a means of access to provision, deploy, and manage the configuration. The BH node serves this purpose.

A bastion host is a host node that belongs to both the private subnet configured by ICM and the public network, and can broker communication between them. To use one, you define a single BH node in your definitions file and set PrivateSubnet to true in your defaults file. For more information, see Deploying on a Private Network.