Where supported, the use of large and huge memory pages can be of significant performance benefit and is highly recommended, as described in the following:
-
IBM AIX® — The use of large pages is highly recommended, especially when configuring over 16GB of shared memory (the sum of the database cache, the routine cache, and the shared memory heaps, as discussed in Estimating Memory Requirements).
By default, when large pages are configured, the system automatically uses them in memory allocation. If shared memory cannot be allocated in large pages, it is allocated in standard (small) pages. However, you can use the memlock parameter for finer-grained control over large pages.
-
Linux (all distributions) — The use of static huge pages (2MB) when available is highly recommended for either physical (bare metal) servers or virtualized servers. Using static huge pages for the InterSystems IRIS shared memory segments yields an average CPU utilization reduction of approximately 10-15% depending on the application.
By default, when huge pages are configured, InterSystems IRIS attempts to provision shared memory in huge pages on startup. If there is not enough space, InterSystems IRIS reverts to standard pages and orphans the allocated huge page space, potentially causing system paging. However, you can use the memlock parameter to control this behavior and fail at startup if huge page allocation fails.
-
Windows — The use of large pages is recommended to reduce page table entry (PTE) overhead.
By default, when large pages are configured, InterSystems IRIS attempts to provision shared memory in large pages on startup. If there is not enough space, InterSystems IRIS reverts to standard pages. However, you can use the memlock parameter to control this behavior and fail at startup if large page allocation fails.
Configuring Large Pages on IBM AIX®
AIX supports multiple page sizes: 4 KB, 64 KB, 16 MB, and 16 GB. Use of 4 KB and 64 KB pages is transparent to InterSystems IRIS. In order for InterSystems IRIS to use 16 MB large pages, you must configure them within AIX. AIX does not automatically change the number of configured large or huge pages based on demand. Currently, InterSystems IRIS does not use 16 GB huge pages.
Large pages should be configured only in high-performance environments because memory allocated to large pages can be used only for large pages.
By default, when large pages are configured, the system automatically uses them in memory allocation. If shared memory cannot be allocated in large pages then it is allocated in standard (small) pages. For finer grain control over large pages, see memlock.
Configure large pages on AIX using the vmo command as follows:
vmo -r -o lgpg_regions=<LargePages> -o lgpg_size=<LargePageSize>
where <LargePages> specifies the number of large pages to reserve, and <LargePageSize> specifies the size, in bytes, of the hardware-supported large pages.
Note:
On systems that support dynamic Logical PARtitioning (LPAR), you can omit the -r option to dynamically configure large pages without a system reboot.
For example, the following command configures 1 GB of large pages:
# vmo -r -o lgpg_regions=64 -o lgpg_size=16777216
Once you have configured large pages, run the bosboot command to save the configuration in the boot image. After the system comes up, enable it for pinned memory using the following vmo command:
vmo -o v_pinshm=1
However, if memlock=64, vmo -o v_pinshm=1 is not required.
Configuring Huge Pages on Linux
On Linux platforms with sufficient huge pages available, the InterSystems IRIS shared memory segment is allocated from the huge page pool. Using huge pages saves memory by saving space in page tables, and allows InterSystems IRIS to lock in its shared memory segment so its pages are not paged out. InterSystems recommends the use of huge pages on systems hosting InterSystems IRIS under most circumstances.
The default memory page size on Linux systems is 4 KB. Most current Linux distributions include an option for huge pages, a memory page size of 2 MB or 1 GB depending on system configuration. When huge pages are configured, InterSystems IRIS automatically uses them in memory allocation.
Important:
With the 2.6.38 kernel, some Linux distributions have introduced transparent huge pages (THP) to automate the creation, management, and use of huge pages. However, THP does not handle the shared memory segments that make up the majority of the InterSystems IRIS memory allocated, and can cause memory allocation delays at runtime that may affect performance, especially for applications that have a high rate of job or process creation. For these reasons, InterSystems recommends that THP be disabled on all systems hosting InterSystems IRIS. For more detailed information on this topic, see Linux Transparent huge pages and the impact to InterSystems IRISOpens in a new tab on InterSystems Developer Community.
To configure huge pages on Linux:
-
Check the status.
/proc/meminfo contains huge pages information. By default, no huge pages are allocated. Default Huge Page size is 2 MB. For example:
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 KB
-
Change the number of huge pages.
You can change the system parameter directly: For example, to allocate 2056 huge pages, execute:
# echo 2056 > /proc/sys/vm/nr_hugepages
Note:
Alternatively, you can use sysctl(8) to change it:
# sysctl -w vm.nr_hugepages=2056
Huge pages must be allocated contiguously, which may require a reboot. Therefore, to guarantee the allocation, as well as to make the change permanent, do the following:
-
Enter a line in /etc/sysctl.conf file:
echo "vm.nr_hugepages=2056" >> /etc/sysctl.conf
-
Reboot the system.
-
Verify meminfo after reboot; for example:
[root woodcrest grub]# tail -4 /proc/meminfo
HugePages_Total: 2056
HugePages_Free: 2056
HugePages_Rsvd: 0
Hugepagesize: 2048 KB
-
Verify the use of huge pages by InterSystems IRIS.
When InterSystems IRIS is started, it reports how much shared memory was allocated; for example, a message similar to the following is displayed (and included in the messages.log file):
Allocated 3580MB shared memory: 3000MB global buffers, 226MB routine buffers
The amount of memory available in huge pages should be greater than the total amount of shared memory to be allocated; if it is not greater, huge pages are not used. For example, with 3580 MB of shared memory and a huge page size of 2 MB, at least 1791 huge pages should be configured.
Note:
It is advisable to set HugePages_Total as close to the shared memory usage as possible, as any unused huge pages are not available to other components.
Configuring Large Pages on Windows
InterSystems recommends enabling InterSystems IRIS to allocate its memory as large pages on Windows systems, which provides more efficient use of memory and paging space. To do so, grant the Windows “Lock Pages in Memory” (SELockMemory) privilege to the Windows user account used to run InterSystems IRIS, which allows InterSystems IRIS to request its memory as large pages. For finer grain control over memory allocation at startup, see memlock.
Note:
If InterSystems IRIS is running under the default SYSTEM account, it has the “Lock Pages in Memory” privilege by default.
When you restart InterSystems IRIS while using large pages, you typically also need to restart Windows to guarantee that all configured memory is allocated. If startup is unable to allocate the full amount of configured memory, it attempts to start up with less memory and may or may not use large pages. You can check the actual memory allocated by reviewing the most recent InterSystems IRIS startup in the messages log.