Ideation for Python Deps Management

# Ideation for Python Deps Management

## Problems

### Orthogonal Levels of Separation: Namespace vs Process

Starting from IPM v0.9.0 or later, ObjectScript packages are separated at the namespace level. For a given package, each namespace can have a different version of it.

Orthogonally, Python packages are *typically* separated at the process level. For a given Python package, each inidividual Python process can have a different version of the package loaded. Once a version of a certain package is loaded in a process, it couldn't be swapped for a different version. There are workarounds which I'll discuss later.

Multiple processes can be running in the same namespace. Conversely, a single process can swtich between namespaces. This orthogonality poses a challenge for correctly manageing Python dependencies along with ObjectScript packages.

### Managing IPM-Installed & User-Installed Python Deps

Users can optionally choose to install Python dependencies manually (without IPM) for their own use purpose by running

``` bash
pip3 install xxx --target /mgr/python
```

This is also what IPM does under the hood.

I wonder if we should **allow management of python packages solely through IPM**. After all, IPM stands for "InterSystems Package Manager", not "ObjectScript Package Manager". It sounds nice to have an `zpm "pip install xxx"` and `zpm "pip uninstall xxx"` command as a convenient wrapper. For example, `zpm "pip install xxx"` will log the installation of `xxx` as a "manual" operation, that shouldn't be uninstalled when running `zpm "uninstall "`

## Ideation of Solutions

### Subinterpreters

At the C level, Python has been supporting *subinterpreters* for a long time. Subinterpreters allow for running multiple (almost) independent interpreters in the same process. Each subinterpreter has its own state, including its own list of imported modules in `sys.modules`. If there is an IRIS-level support for switching between subinterpreters via an ObjectScript call, we can have multiple "parallel" interpreters in the same process. Each will be configured to have a slightly different `sys.path`, allowing for a different version of the target python package for import.

Here's an minimum proof-of-concept

```
.
├── bar
│   └── mypackage.py
├── foo
│   └── mypackage.py
└── main.c

```

```python
# foo/mypackage.py
__version__ = "1.0.0"
```

```python
# bar/mypackage.py
__version__ = "2.0.0"
```

```c
// main.c
#include
#include

int main(int argc, char *argv[]) {
Py_Initialize();

PyThreadState *subinterp1 = Py_NewInterpreter();
PyRun_SimpleString(
"import sys\n"
"sys.path.insert(0, 'foo')\n"
"import mypackage\n"
"print('Sub-interpreter 1 imported version', mypackage.__version__)\n"
);
Py_EndInterpreter(subinterp1);

PyThreadState *subinterp2 = Py_NewInterpreter();
PyRun_SimpleString(
"import sys\n"
"sys.path.insert(0, 'bar')\n"
"import mypackage\n"
"print('Sub-interpreter 2 imported version', mypackage.__version__)\n"
);
Py_EndInterpreter(subinterp2);

Py_Finalize();
return 0;
}

// Note on compilation:
// You can find the include paths using `python3-config --includes` and the dynamic library paths using `python3-config --ldflags`. On my MacBook pro with Python 3.13 installed from homebrew, the command I used to compile is:
// gcc main.c -o main -I/opt/homebrew/opt/python@3.13/Frameworks/Python.framework/Versions/3.13/include/python3.13 -I/opt/homebrew/opt/python@3.13/Frameworks/Python.framework/Versions/3.13/include/python3.13 -L/opt/homebrew/opt/python@3.13/Frameworks/Python.framework/Versions/3.13/lib/python3.13/config-3.13-darwin -lpython3.13
```

The output of running the above program is

```
Sub-interpreter 1 imported version 1.0.0
Sub-interpreter 2 imported version 2.0.0
```

Apparently, this requires significant help from the Data Platforms team. What we want is 3 methods (or $zu(xxx) functions) exposed at the ObjectScript level:

* one that creates a subinterpreter and switches to it (a wrapper around `Py_NewInterpreter()`),
* one that switches between subinterpreters (a wrapper around `PyThreadState_Swap()`), and
* one that ends a subinterpreter (a wrapper around `Py_EndInterpreter()`).

### Save Packages in Namespace-Specific Directory

It makes sense that each namespace has its own set of python dependencies, the same way each namespace has its own database for ObjectScript code although we allow package and routine mapping.

This works nicely either with subinterpreters or under the assumption that each process will only run one IPM package and doesn't change namespace. Notice that we need to manually manipulate `sys.path` to include the namespace-specific directory for packages.

### Unify Management of Packages

IPM can have a centralized management of packages, either through a `requirements.txt` on filesystem or preferably a `%Persistent` class. The `%Persistent` class keeps track of the dependency relationship between Python packages and ObjectScript packages.

For every IPM package, we record its set of Python dependencies. We combine it with the accumulated list of Python dependencies from previous installations of other IPM packages. We then use `pip-compile` (an open-source tool) to compute a new list of dependencies and use `pip-sync` (also part of the same tool) to synchronize the changes to dependencies. Somehow, I didn't see an option to specify the target path in `pip-sync` in the doucmentation though.

### More ideas to come ...