google / google/symphony-gcp

Proposal: upgrade hf-provider from Python 3.9 to 3.12

Open
#103 0 comments 0 reactions 0 assignees View on GitHub
dependencies python
Dominant language
Python
Stars
7
Forks
2
PR merge metrics
No merged PRs in 30d

Description

## Summary
Following up on team discussions about this, creating a GitHub issue to track and document so we can decide in one place.

hf-provider is pinned to Python 3.9, which reached end-of-life in October 2025. The latest releases of most of our runtime dependencies now require Python 3.10 or newer, so we can't pick up newer versions of those libraries while we stay on 3.9.

Nothing is broken today, and nothing forces a change. This proposal lays out what an upgrade to Python 3.12 would involve, so we can decide together whether and when it's worth doing.

## Current state
- hf-provider requires Python `==3.9.*` (`hf-provider/pyproject.toml`).
- With the current pins, everything builds, tests pass, and there are no warnings.
- When upgrading dependencies to recent versions, they emitted Python 3.9 EOL deprecation warnings and the latest releases of most of them now exclude 3.9 entirely (table below).

## Dependency status (PyPI, 2026-07-16)

| Package | Our pin | Last 3.9-compatible | Latest (requires Python) |
|----------------------|-----------|---------------------|--------------------------|
| kubernetes | ==32.0.1 | 35.0.0 | 36.0.3 (>=3.10) |
| google-cloud-compute | >=1.31.0 | 1.47.0 | 1.49.0 (>=3.10) |
| google-cloud-pubsub | >=2.30.0 | 2.38.0 | 2.39.0 (>=3.10) |
| google-auth | >=2.40.3 | 2.50.0 | 2.56.0 (>=3.10) |
| tenacity | >=9.1.2 | 9.1.2 (= our pin) | 9.1.4 (>=3.10) |
| python-dotenv | ==1.1.0 | 1.2.1 | 1.2.2 (>=3.10) |
| importlib-metadata | >=8.7.0 | 8.7.1 | 9.0.0 (>=3.10) |
| pytest | >=7.0.0 | 8.4.2 | 9.1.1 (>=3.10) |

pydantic, PyYAML, typing-extensions, python-debouncer, and PyInstaller still support 3.9 in their latest releases.

In practice this means that on 3.9, pip can give us at most the "last 3.9-compatible" version of each package above. Anything released after that point, including security fixes, isn't installable for us without the interpreter upgrade.

## Impact on RHEL / Rocky Linux
- The CLIs ship as PyInstaller binaries with the interpreter bundled, so hosts running the RPM binaries wouldn't need Python 3.12 installed. Nothing changes for them.
- The change is in our CI build containers: `build_rpm.yaml` currently installs `python39` on RockyLinux 8 and 9. A `python3.12` package is available on RL8 (8.10+) and RL9, and it's already RL10's default `python3`.
- glibc compatibility comes from building inside the per-version RL containers, and that part stays exactly as it is.

## Proposed plan
- [ ] PR 1: hf-provider, bump `requires-python` to 3.12, update the `build_rpm.yaml` matrix, and re-verify the PyInstaller specs and RPM install on RL8/9/10. Dependencies change only as much as needed to build.
- [ ] PR 2: hf-provider, upgrade dependencies to current versions.

Why 3.12 rather than 3.10 or 3.11: Python 3.10 reaches end-of-life this October, RL10 already ships 3.12 as its default, and k8s-operator already requires 3.12, so both packages would end up on the same version. (https://devguide.python.org/versions/)

## Validation
1. The existing pytest suites already run inside the RL8/9/10 build containers on every push, so they'd cover 3.12 automatically once the matrix changes.
2. The frozen binaries deserve extra attention: #39, #60, and #89 were all failures that only showed up in the PyInstaller binaries or the published RPM while unit tests passed. So before merging, the idea is to install the built RPMs on each RL version and run the five HostFactory commands against a fixture config, checking JSON output and exit codes. We could also add a small version of that check to CI while we're at it.
3. Full end-to-end validation in our Symphony + GCP test environment. HostFactory invoking the upgraded binaries through the real entry points with a request/return cycle on both the GCE and GKE path.

## What stays the same
- How Symphony talks to the providers: the five HostFactory commands keep the same JSON input/output and exit codes. The upgrade swaps the interpreter and libraries underneath, not the behavior.
- The supported OS versions: RPMs keep being built for RockyLinux 8, 9, and 10, same as today.
- Anything beyond the upgrade itself: no feature work or refactoring would ride along in these PRs, the diffs contain only what the version changes require.

## If we decide not to do it now
That's a valid outcome too. We'd stay on the last 3.9-compatible versions of the packages above, and anything released after those, including security fixes, would stay out of reach until we upgrade. If we go this way, it might help to agree on what would make us revisit, for example a security advisory affecting one of these libraries.

Contributor guide

Open the contributing guide

Research direction

Start by reading hf-provider/pyproject.toml and build_rpm.yaml, then inspect the existing pytest suites in the Rocky Linux 8/9/10 build containers. Verify the PyInstaller specs and install each built RPM on RL8/9/10, running the five HostFactory commands against a fixture config and checking JSON output and exit codes. Done means the 3.12 matrix builds without behavior changes, followed by the described Symphony + GCP end-to-end check.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, ci-cd, release
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.