DataTalksClub / DataTalksClub/website

D1.1 Replace DTC jobs with the package jobs app (relay backend)

Open
#349 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement foundation P1
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Plan issue: community-base docs/plan/phase-1.md D1.1. Depends on C1.5, R1.1, R1.2 (all done); the D0.1a package pin (v0.3.0) is on main.

Goal

Replace the site's jobs app (DurableJob machinery on django_q) with the community-base jobs app, executed by Relay.

Scope

  • Install community_base.jobs (P2). Re-point every jobs.registry.register_handler and jobs.dispatch call site to the package, keeping handler names.
  • Move schedule semantics to the package: the package's cb-jobs-run-due / cb-jobs-sweep Relay schedules drive due intents; the site's dtc:durable-job-relay django_q bridge dies with django_q.
  • Data migration (P6): copy pending jobs.DurableJob rows into cb_jobs.JobIntent (raw-SQL migration in a surviving app so it runs after the old app is removed); old tables stay for a later cleanup.
  • Settings: COMMUNITY_BASE["SITE_URL"], RELAY_BASE_URL, RELAY_API_KEY, RELAY_WEBHOOK_SECRET read from the environment (deployment secrets provision the values); JOBS_BACKEND="relay" in development/production, "sync" in test/local.
  • Add sync_relay_schedules to the deploy steps after migrate; remove the qcluster container from deploy task definitions.
  • Remove django_q from INSTALLED_APPS (+ Q_CLUSTER), remove the jobs/ app and its tests; mount the package ingress route and allow internal/jobs/run in the core.middleware request boundary (CSRF exempt, no session).

Non-goals

  • Handler business logic does not change — only registration and transport.
  • Dropping the orphaned jobs_durablejob / django_q tables (later cleanup once the copy is verified in development).
  • Provisioning the Relay client secret in the dev environment (operator step; the code reads RELAY_API_KEY from env and dispatch failures are retried until it is set).

Acceptance criteria

  • uv run pytest -q passes (full suite)
  • uv run python manage.py makemigrations --check --dry-run reports no changes beyond the intended migration
  • grep -rn "django_q\|from jobs" --include=*.py website/ events/ email_app/ content_sync/ api/ shows no stale runtime imports outside the historical migration
  • Development deploy: manage.py jobs_ingress_selftest -> OK and sync_relay_schedules --dry-run -> no diff (needs the operator-provisioned Relay client secret; recorded here when run)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read the D1.1 plan reference in docs/plan/phase-1.md, then trace jobs.registry.register_handler and jobs.dispatch call sites, the jobs app, settings, deploy task definitions, and core.middleware. Start by mapping the package integration and raw-SQL data migration dependencies. Done means the full pytest and migration checks pass, no stale runtime imports remain, and the deployment self-test and schedule sync meet the stated acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
django, python
Domain
backend, database, devops
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.