Modifying Provisioned Infrastructure

At any time following successful provisioning — including after successful services deployment using the icm run command — you can alter the provisioned infrastructure or configuration by modifying your definitions.json file and executing the icm provision command again. If changing a deployed configuration, you would then execute the icm run command again, as described in Redeploying Services.

You can modify existing infrastructure or a deployed configuration in the following ways.

To change the characteristics of one or more nodes, change settings within the node definitions in the definitions file. You might want to do this to vertically scale the nodes; for example, in the following definition, you could change the DataVolumeSize setting (see General Parameters) to increase the sizes of the DM nodes’ data volumes:
```
{
    "Role": "DM",
    "Count": "2",
    "LicenseKey": "standard-iris.key”,
    "InstanceType": "m4.xlarge",
    "OSVolumeSize": "32",
    "DataVolumeSize": "25"
 },
```
Caution:

Modifying attributes of existing nodes such as changing disk sizes, adding CPUs, and so on may cause those nodes (including their persistent storage) to be recreated. This behavior is highly specific to each cloud provider, and caution should be used to avoid the possibility of corrupting or losing data.

Important:

Changes to the Label and Tag fields in the definitions.json file are not supported when reprovisioning.
To add nodes, modify the definitions.json file in one or more of the following ways:
- Add a new node type by adding a definition. For example, if you have deployed a sharded cluster with data nodes only, you can add compute nodes by adding an appropriate COMPUTE node definition to the definitions file.
- Add more of an existing node type by increasing the Count specification in its definition. For example, to add two more application servers to a distributed cache cluster that already has two, you would modify the AM definition by changing “Count”: “2” to “Count”: “4”. When you add nodes to existing infrastructure or a deployed configuration, existing nodes are not restarted or modified, and their persistent storage remains intact.
  
  Note:
  
  When you add data nodes to a deployed sharded cluster after it has been loaded with data, you can automatically redistribute the sharded data across the new servers (although this must be done with the cluster offline); for more information, see Add Shard Data Servers and Rebalance DataOpens in a new tab in the “Horizontally Scaling InterSystems IRIS for Data Volume with Sharding” chapter of the Scalability Guide.
  Generally, there are many application-specific attributes that cannot be modified by ICM and must be modified manually after adding nodes.
- Add a load balancer by adding “LoadBalancer": "true" to DATA, COMPUTE, AM, or WS node definitions.
To remove nodes, decrease the Count specification in the node type definition. To remove all nodes of a given type, reduce the count to 0.

Caution:

Do not remove a definition entirely; Terraform will not detect the change and your infrastructure or deployed configuration will include orphaned nodes that ICM is no longer tracking.

Important:

When removing one or more nodes, you cannot choose which to remove; rather nodes are unprovisioned on a last in, first out basis, so the most recently created node is the first one to be removed. This is especially significant when you have added one or more nodes in a previous reprovisioning operation, as these will be removed before the originally provisioned nodes.

You can also remove load balancers by removing “LoadBalancer": "true" from a node definition, or changing the value to false.

There are some limitations to modifying an existing configuration through reprovisioning, as follows:

You cannot remove nodes that store data — DATA, DM, or DS.
You cannot change a configuration from nonmirrored to mirrored, or the reverse.
You can add DATA, DS, or DM nodes to a mirrored configuration only in numbers that match the relevant MirrorMap setting, as described in Rules for Mirroring. For example, if the MirrorMap default value of primary,backup is in effect, DATA and DS nodes can be added in even numbers (multiples of two) only to be configured as additional failover pairs, and DM nodes cannot be added; if MirrorMap is primary,backup,async, DATA or DS nodes can be added in multiples of three to be configured as additional three-member mirrors, or as a pair to be configured as an additional mirror with no async, and a DM node can be added only if the existing DM mirror does not currently include an async.
You can add an arbiter (AR node) to a mirrored configuration, but it must be manually configured as arbiter, using the management portal or ^MIRROR routine, for each mirror in the configuration.

By default, when issuing the icm provision command to modify existing infrastructure, ICM prompts you to confirm; you can avoid this, for example when using a script, by using the -force option.

Remember that after reprovisioning a deployed configuration, you must issue the icm run command again to redeploy.

Overcoming Provisioning Errors