AbsaOSS / AbsaOSS/EventGate

Upgrade the Lambda runtime from Python 3.13 to 3.14

Đang mở
#222 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
infrastructure type:tech-debt
Ngôn ngữ chính
Python
Star
4
Fork
0
Merge trung bình
1 ngày 5 giờ
Pull request đã merge (30 ngày)
8

Mô tả

### Description of Technical Debt

The local toolchain and the deployed runtime are on different Python versions:

| Where | Version | Source |
|---|---|---|
| Local venv / test runs | 3.14.7 | `./.venv/Scripts/python.exe --version` |
| Container image | 3.13 | `Dockerfile`: `public.ecr.aws/lambda/python:3.13-arm64` |
| mypy target | 3.13 | `pyproject.toml` `[tool.mypy] python_version` |
| black target | py313 | `pyproject.toml` `[tool.black] target-version` |

Every local `pytest`, `mypy`, `pylint`, and `black` run therefore executes under a different interpreter than production, and nothing in CI closes the gap.

Surfaced during review of PR #204, where the difference in annotation semantics between the two versions came up directly.

### Impact of Technical Debt

- **Local green does not prove production green.** Anything that changed between 3.13 and 3.14 passes locally and fails on deploy. Annotation evaluation is the concrete example: PEP 649 makes annotations lazy by default in 3.14 but not in 3.13, so a forward reference that resolves on a developer machine can raise `NameError` in Lambda.
- **The static analysis is checking the wrong target.** mypy is told `python_version = "3.13"` while running on a 3.14 interpreter, so its view of the stdlib and the developer's actual runtime disagree.
- **Review time is spent on it.** The 3.13-vs-3.14 distinction already consumed a PR #204 thread, and will keep resurfacing while the two differ.
- **The divergence widens on its own.** Nothing pins the local version, so a fresh venv on a newer interpreter moves further from production without anyone noticing.

### Category

DevOps / Infrastructure

### Priority

Medium - Should be addressed soon

### Proposed Solution

Move the deployed runtime to 3.14 so it matches what the project is already developed and tested on. The code already runs there — 290 unit tests pass locally on 3.14.7 — so this is a deployment-target change rather than a migration.

Verified against ECR Public and PyPI:

- `public.ecr.aws/lambda/python` publishes GA arm64 builds for 3.14 — 37 non-preview arm64 tags, latest `3.14.2026.08.26.13-arm64`.
- There is **no floating `3.14-arm64` tag**, unlike `3.13-arm64`. The plain `3.14` tag is a single-arch manifest, not a multi-arch list, so it cannot stand in for an arm64 build. The Dockerfile must pin a dated arm64 tag.
- Compiled dependencies publish cp314 artifacts: `confluent-kafka==2.15.0` (5 cp314 wheels), `psycopg2==2.9.12` (cp314 wheel, 3.14 classifier), `cryptography==50.0.0` (13 cp314 wheels, 3.14 classifier). `aws-lambda-powertools==3.31.1` declares the 3.14 classifier.

Work:

1. `Dockerfile`: `FROM --platform=linux/arm64 public.ecr.aws/lambda/python:3.14.-arm64`.
2. `pyproject.toml`: `[tool.mypy] python_version = "3.14"`, `[tool.black] target-version = ['py314']`.
3. Rebuild the image and confirm the source builds still succeed. This is the main risk: the Dockerfile passes `--no-binary confluent-kafka` to compile against the librdkafka 2.15.0 built in the image, so the published cp314 wheels are **not** used and the C extension is compiled against the 3.14 C API directly. `psycopg2` (not `psycopg2-binary`) compiles from source too.
4. Confirm the remaining pure-Python pins install cleanly on 3.14: `jsonschema`, `PyJWT`, `requests`, `boto3`/`botocore`, `aiosql`.
5. Run the integration tests against the rebuilt image, not just unit tests, so the Kafka and Postgres paths exercise the recompiled extensions.
6. Check whether CI pins a Python version anywhere and update it, so the mismatch cannot silently reappear.

**Alternative:** keep the runtime on 3.13 and recreate the venv on 3.13 instead, pinning the version in CI. That closes the gap equally well and avoids a runtime bump, at the cost of developing against an older interpreter. Either direction is acceptable; leaving the two apart is not.

### Effort Estimate

1–2 days, dominated by rebuilding the image and validating the compiled extensions

### Dependencies / Related

- PR #204, where this surfaced
- #193

### Additional Context

Done when:

- The image builds on 3.14 with librdkafka and psycopg2 compiled from source.
- Unit and integration tests pass against the rebuilt image.
- No tooling config still names 3.13.
- A deployed smoke test covers one `POST /topics/{topic_name}` fan-out and one `/health` probe.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.