bcgov / bcgov/lcfs

TFRS - Dependency Updates and Pipeline Upgrade

Open
#4,416 0 comments 0 reactions 1 assignee Claimed by @AlexZorkin View on GitHub
High Tech Debt
Dominant language
Python
Stars
6
Forks
6
Avg merge
2d 7h
Merged PRs (30d)
87

Description

**Describe the task**

Complete the TFRS backend's Django 3.2 → 4.2 upgrade on `release-3.0.1`: bring Python dependencies up to Django-4.2-compatible versions, fix source-level uses of removed Django APIs, restore the deployment pipeline's manual-approval flow, and resolve the Postgres 15 schema-permission blocker preventing prod migrations from completing.

**Purpose**

Dependabot bumped Django 3.2 → 4.2 on `release-3.0.1`, but the surrounding ecosystem (django-extensions, DRF, celery, db_comments patch, several pinned packages) wasn't moved with it. The backend pod won't start in any environment until the cascade is resolved. Several dependabot bumps also need to be rolled back because they require Python 3.10+ on a 3.9 image. Separately, prod migrations now fail at `django_celery_beat.0011` with `permission denied for schema public` because Postgres 15 revoked default `CREATE` on `public`, and the new `django-celery-beat==2.5.0` is the first thing in a while to try creating new tables. Finally, the deployment approval workflow lists two ex-team-members who can't be assigned to issues, blocking every test/prod deploy at the approval step.

**Acceptance Criteria**

- [ ] `backend/db_comments/patch_fields.py` made Django 4.2 compatible (db_comment setter added, kwarg no longer stripped).
- [ ] All `django.conf.urls.url` usages in project source replaced with `django.urls.re_path`.
- [ ] Bump to versions supporting Django 4.2 + Python 3.9: `django-extensions`, `djangorestframework`, `django-celery-beat`, `celery`, `kombu`, `amqp`, `billiard`, `vine`, `cryptography`, `cffi`, `typing_extensions`.
- [ ] Roll back to cp39-compatible versions: `python-dotenv`, `requests`, `urllib3`.
- [ ] Pin `setuptools<81` to keep `pkg_resources` for `coreapi`.
- [ ] Remove `JulianForeman` and `dhaselhan` from approver lists in `test-ci.yaml` and `prod-ci.yaml`.
- [ ] Run `GRANT CREATE, USAGE ON SCHEMA public TO ` (and `ALTER DEFAULT PRIVILEGES ...`) on the prod database so `django-celery-beat` migrations can complete. Verify the same is in place on test.
- [ ] Backend pod starts past the `dbmigration` init container in dev, test, and prod.
- [ ] Verify `/api/notifications/count`, `/api/users/current`, IDIR + BCeID login, and the `/api/doc/` Swagger endpoint all work end-to-end in test and prod after the deploy.

**Additional context**

- Backend image is `python:3.9.20-bullseye` (upstream EOL). Future dependabot bumps will keep tripping the 3.10+ floor — bumping the base image to 3.11/3.12 should be the next ticket.
- `coreapi` (last release 2018) is the only thing keeping the `setuptools<81` cap. Migrating `/api/doc/` to `drf-spectacular` (or removing it) would let us drop the cap.
- Other Django 3.x-era pins survive `manage.py check` but remain brittle: `django-filter==2.4.0`, `django-debug-toolbar==3.2.4`, `djangorestframework-bulk==0.2.1`, `django-nose==1.4.7`. Worth a coordinated hygiene pass.
- Django's default LOGGING swallows view tracebacks when `DEBUG=False` because the console handler is gated on `require_debug_true`. A small LOGGING block in `settings.py` would have saved hours of diagnosis time on the `/api/notifications/count` 500 — worth a separate observability ticket.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.