influxdata / influxdata/influxdb
Define process and implement venv migration for python minor release upgrades
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
This attempts to help define the TBD processes outlined in https://github.com/influxdata/influxdb/blob/main/README_processing_engine.md#how-will-influxdata-maintain-the-embedded-interpreter. It isn't strictly needed for GA since the code could be added at a later date. Updating to 3.13 prior to beta (see https://github.com/influxdata/influxdb/issues/26044) will provide even more breathing room. UPDATE: we updated to 3.13 before GA.
https://devguide.python.org/versions/ lists the support dates. Currently, 3.11 will receive support until 2027-10 and 3.13 until 2029-10.
Python uses MAJOR.MINOR.MICRO versioning. When a venv is created, it keeps a record of how the venv was created in `/path/to/venv/pyvenv.cfg`. Eg:
```
# Linux/OSX
$ cat /path/to/plugins/.venv/pyvenv.cfg
home = /path/to/influxdb3/install/python/bin
include-system-site-packages = false
version = 3.11.11
executable = /path/to/influxdb3/install/python/bin/python3.11
command = /path/to/influxdb3/install/python/bin/python3 -m venv /path/to/plugins/.venv
# Windows
$ type \path\to\plugins\.venv\pyvenv.cfg
home = \path\to\influxdb3\install\python
include-system-site-packages = false
version = 3.11.11
executable = \path\to\influxdb3\install\python\python.exe
command = \path\to\influxdb3\install\python\python.exe -m venv \path\to\plugins\.venv
```
The details are discussed in https://peps.python.org/pep-0405/, but importantly, `influxdb3` should pay attention to the `version` field (NOT `executable` since it doesn't encode the version on windows) and the in use python runtime. When MAJOR.MINOR is the same, the venv is expected to work and we shouldn't have to do anything, but when different, we need to take action.
A straw person on how to handle this might be:
1. on initial creation and install package, run `python3 -n pip freeze > requirements.txt`
* how to handle/detect manual pip installs? Perhaps unconditionally run `python3 -n pip freeze > requirements.txt` prior to venv activation?
2. on upgrade from different minors (eg, 3.11.11 to 3.11.12), do nothing
* perhaps we want to log at INFO or DEBUG?
3. It's possible to do an in place upgrade from different minors (eg, 3.11 to 3.13). Eg:
```
# pseudocode
warn!("Detected new python version, backing up .venv to .venv-3.11.zip and migrating to 3.13")
mv /path/to/plugins/.venv /path/to/plugins/.venv-3.11 # 3.11 calculated from 'version' in pyvenv.cfg
python3 -m venv /path/to/plugins/.venv # using new python3
cp /path/to/plugins/.venv-3.11/requirements.txt /path/to/plugins/.venv
python3 -m pip -r /path/to/plugins/.venv-3.11/requirements.txt # using new python3
zip /path/to/plugins/.venv-3.11
warn!("Migration complete. Please test to ensure functionality.")
```
4. We should have clear instructions for how users can test functionality prior to upgrades. Eg:
a. install new `influxdb3` to a different location
b. copy the production 'plugins' directory (and venv if separate) to a new location
c. point the new `influxdb3 serve` invocation to the new 'plugins' directory (and venv if separate)
d. test functionality, adjusting scripts and requirements.txt as necessary
e. if testing required no changes, perform an upgrade like normal; otherwise, shutdown influxdb3, perform upgrade, copy updated scripts to 'plugins' and copy updated requirements.txt to venv, perform upgrade
The details of '3' and '4' of course may need to be updated from the above, but that is the general idea. Details for handling alternative venvs (ie, `--virtual-env-location`) intentionally omitted.
Contributor guide
Research direction
Start with the embedded interpreter process in README_processing_engine.md and the pyvenv.cfg version details described in the issue. Trace how influxdb3 serve handles the plugins directory and --virtual-env-location. Done would require an agreed migration and testing process plus implementation, but the issue names no source files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100