Use the Flexible Python Runtime Feature
Overview of the Flexible Python Runtime Feature
When you run Embedded Python, InterSystems IRIS expects that you are using the default version of Python for your operating system. However, there are times when you might want to upgrade to a later version of Python or switch to an alternate distribution like Anaconda. The Flexible Python Runtime feature enables you to use these versions of Python with Embedded Python in InterSystems IRIS.
The Flexible Python Runtime feature is not supported on all operating systems. See Other Supported Features for a complete list of platforms that support the feature.
Preparing to use the Flexible Python Runtime feature involves three basic steps:
-
Install the version of Python you want to use.
-
Set the PythonRuntimeLibrary configuration setting to specify the location of the Python runtime library to use when running Embedded Python.
For example:/usr/lib/x86_64-linux-gnu/libpython3.11.so.1
This location will vary based on your operating system, Python version, and other factors. The example shown here is for Python 3.11 on Ubuntu 22.04 on the x86 architecture.
See PythonRuntimeLibrary for more information.
-
Ensure that the sys.path variable in Embedded Python includes the correct directories needed to import Python packages.
Embedded Python and sys.path
After you launch Embedded Python, it uses the directories contained in the sys.path variable to locate any Python packages you want to import.
When you use Embedded Python with the default version of Python for your operating system, sys.path already includes the correct directories, for example.
-
<installdir>/lib/python (Python packages reserved for InterSystems IRIS)
-
<installdir>/mgr/python (Python packages installed by the user)
-
Global package repositories for the default Python version
For example: /usr/local/lib/python3.10/dist-packages.
This location will vary based on your operating system, Python version, and other factors. The example shown here is for Python 3.10 on Ubuntu 22.04.
On Ubuntu 22.04, with the default version of Python (3.10), sys.path in Embedded Python might look something like this:
USER>do ##class(%SYS.Python).Shell()
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type quit() or Ctrl-D to exit this shell.
>>> import sys
>>> sys.path
['/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/InterSystems/IRIS/lib/python',
'/InterSystems/IRIS/mgr/python', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages',
'/usr/lib/python3.10/dist-packages']
If sys.path contains a directory in that is within your home directory, such as /home/<user>/.local/lib/python3.10/site-packages, it could indicate that you have installed packages in your local package repository. For example, if you install a package without the --target attribute (and without using sudo), Python will install it into the local package repository within your home directory. If any other user tries to import the package, it will fail.
Although InterSystems recommends using the --target <installdir>/mgr/python option, installing packages using sudo and omitting the --target attribute installs the packages into the global package repository. These packages can also be imported by any user.
If you switch to an alternate distribution like Anaconda, InterSystems IRIS may not know where its package repositories are located. InterSystems IRIS provides you with a tool that can help you tailor your sys.path to include the correct directories, namely the iris_site.py file in the directory <installdir>/lib/python.
For Ubuntu 22.04, when using Anaconda, edit your iris_site.py file to look something like the following:
import sys
from site import getsitepackages as __sitegetsitepackages
# modify EmbeddedPython to get site-packages from Anaconda python distribution.
def set_site_path(platform_name):
sys.path = sys.path + __sitegetsitepackages(['/opt/anaconda3/'])
With the iris_site.py above, sys.path in Embedded Python might look something like this:
USER>do ##class(%SYS.Python).Shell()
Python 3.11.7 (main, Dec 15 2023, 18:24:52) [GCC 11.2.0] on linux
Type quit() or Ctrl-D to exit this shell.
>>> import sys
>>> sys.path
['/opt/anaconda3/lib/python311.zip', '/opt/anaconda3/lib/python3.11', '/opt/anaconda3/lib/python3.11/lib-dynload',
'/InterSystems/IRIS/lib/python', '/InterSystems/IRIS/mgr/python', '/opt/anaconda3/lib/python3.11/site-packages']
It may take you a few iterations to get your sys.path correct for Flexible Python Runtime. It may be helpful to launch Python outside of InterSystems IRIS and compare its sys.path with the sys.path inside Embedded Python to make sure you have all of the expected directories.
Changes to the PythonRuntimeLibrary configuration setting or iris_site.py take effect on starting a new session. Restarting InterSystems IRIS is not required.
Once you have gotten your iris_site.py file the way you want it, copy it to the <installdir>/mgr/python directory. This will ensure that your configuration is not wiped out when you upgrade to a later version of InterSystems IRIS.
Check Python Version Information
If you are using the Flexible Python Runtime feature, it can be useful to check the version of Python that Embedded Python is using versus the default version that your system is using. An easy way to do this is by calling the GetPythonInfo() method of the %SYS.PythonOpens in a new tab class.
The following example, on Ubuntu 20.04 on the x86 architecture, shows that the Python runtime library being used is /usr/lib/x86_64-linux-gnu/libpython3.10.so.1, the running version of Embedded Python is 3.10.13, and the system version is 3.8.10.
USER>do ##class(%SYS.Python).GetPythonInfo(.info)
USER>zw info
info("CPF_PythonPath")=""
info("CPF_PythonRuntimeLibrary")="/usr/lib/x86_64-linux-gnu/libpython3.10.so.1"
info("RunningLibrary")="/usr/lib/x86_64-linux-gnu/libpython3.10.so.1"
info("RunningVersion")="3.10.13 (main, Aug 25 2023, 13:20:03) [GCC 9.4.0]"
info("SystemPath")="/usr/lib/python3.8/config-3.8-x86_64-linux-gnu"
info("SystemVersion")="3.8.10 (default, Nov 14 2022, 12:59:47) [GCC 9.4.0]"
info("SystemVersionShort")="3.8.10"
This information will vary based on your operating system, Python version, and other factors.
Flexible Python Runtime Example: Python 3.11 on Ubuntu 22.04
Python 3.10 is the default version of Python on Ubuntu 22.04. This example shows how to use Python 3.11 with Embedded Python.
This example is for Ubuntu 22.04 on the x86 architecture. File and directory names may vary if you are on the ARM architecture.
-
Install Python 3.11 from the command line.
$ sudo apt install python3.11-full
-
Install the Python 3.11 libpython.so shared library.
$ sudo apt install libpython3.11
-
Set the PythonRuntimeLibrary configuration setting to /usr/lib/x86_64-linux-gnu/libpython3.11.so.1.
See PythonRuntimeLibrary for more information.
-
From Terminal, launch Embedded Python and verify that sys.path now includes the Python 3.11 package directories.
USER>do ##class(%SYS.Python).Shell() Python 3.11.0rc1 (main, Aug 12 2022, 10:02:14) [GCC 11.2.0] on linux Type quit() or Ctrl-D to exit this shell. >>> import sys >>> sys.path ['/usr/lib/python311.zip', '/usr/lib/python3.11', '/usr/lib/python3.11/lib-dynload', '/InterSystems/IRIS/lib/python', '/InterSystems/IRIS/mgr/python', '/usr/local/lib/python3.11/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.11/dist-packages'] >>>
-
From Terminal, use the GetPythonInfo() method of the %SYS.PythonOpens in a new tab class to view the Python version information.
USER>do ##class(%SYS.Python).GetPythonInfo(.info) USER>zw info info("AllowNonSystemPythonForIntegratedML")=0 info("CPF_PythonPath")="" info("CPF_PythonRuntimeLibrary")="/usr/lib/x86_64-linux-gnu/libpython3.11.so.1" info("RunningLibrary")="/usr/lib/x86_64-linux-gnu/libpython3.11.so.1" info("RunningVersion")="3.11.0rc1 (main, Aug 12 2022, 10:02:14) [GCC 11.2.0]" info("SystemPath")="/usr/lib/python3.10/config-3.10-x86_64-linux-gnu" info("SystemVersion")="3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0]" info("SystemVersionShort")="3.10.6"
This example shows that the Python runtime library being used is /usr/lib/x86_64-linux-gnu/libpython3.11.so.1, the running version of Embedded Python is 3.11.0rc1, and the system version is 3.10.6.
Flexible Python Runtime Example: Anaconda on Ubuntu 22.04
Anaconda is a Python-based platform commonly used for data science and artificial intelligence applications. This example shows how to use Anaconda with Ubuntu 22.04.
This example is for Ubuntu 22.04 on the x86 architecture. File and directory names may vary if you are on the ARM architecture.
-
Download Anaconda from https://www.anaconda.com/download.
The download will be a shell script with a name similar to Anaconda3-2024.02-1-Linux-x86_64.sh.
-
Run the Anaconda installation script from the command line, for example:
$ sudo sh Anaconda3-2024.02-1-Linux-x86_64.sh -b -p /opt/anaconda3
-
Set the PythonRuntimeLibrary configuration setting to /opt/anaconda3/lib/libpython3.11.so.
See PythonRuntimeLibrary for more information.
-
Edit the iris_site.py file in the directory <installdir>/lib/python to add the Anaconda package repository to sys.path:
import sys from site import getsitepackages as __sitegetsitepackages # modify EmbeddedPython to get site-packages from Anaconda python distribution. def set_site_path(platform_name): sys.path = sys.path + __sitegetsitepackages(['/opt/anaconda3/'])
-
If you launch Embedded Python and see the error <class 'ModuleNotFoundError'>: No module named 'math' - iris loader failed, edit your PATH environment variable to add the directory /opt/anaconda3/bin to the front:
$ export PATH=/opt/anaconda3/bin:$PATH $ env |grep PATH PATH=/opt/anaconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin
-
From Terminal, launch Embedded Python and verify that sys.path now includes the Anaconda package repositories.
USER>do ##class(%SYS.Python).Shell() Python 3.11.7 (main, Dec 15 2023, 18:24:52) [GCC 11.2.0] on linux Type quit() or Ctrl-D to exit this shell. >>> import sys >>> sys.path ['/opt/anaconda3/lib/python311.zip', '/opt/anaconda3/lib/python3.11', '/opt/anaconda3/lib/python3.11/lib-dynload', '/InterSystems/IRIS/lib/python', '/InterSystems/IRIS/mgr/python', '/opt/anaconda3/lib/python3.11/site-packages']
-
From Terminal, use the GetPythonInfo() method of the %SYS.PythonOpens in a new tab class to view the Python version information.
USER>zw info info("AllowNonSystemPythonForIntegratedML")=0 info("CPF_PythonPath")="" info("CPF_PythonRuntimeLibrary")="/opt/anaconda3/lib/libpython3.11.so" info("RunningLibrary")="/opt/anaconda3/lib/libpython3.11.so" info("RunningVersion")="3.11.7 (main, Dec 15 2023, 18:24:52) [GCC 11.2.0]" info("SystemPath")="/usr/lib/python3.10/config-3.10-x86_64-linux-gnu" info("SystemVersion")="3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0]" info("SystemVersionShort")="3.10.6"
This example shows that the Python runtime library being used is /opt/anaconda3/lib/libpython3.11.so, the running version of Embedded Python is 3.11.7, and the system version is 3.10.6.
Once you have gotten your iris_site.py file the way you want it, copy it to the <installdir>/mgr/python directory. This will ensure that your configuration is not wiped out when you upgrade to a later version of InterSystems IRIS.