E2E smoke test on dev after deploy (Playwright) + Datamailer email verification

Goal

Add an automated end-to-end smoke test that runs against dev after every deploy. It provisions a full course lifecycle (course → homework → project → submissions → scoring → leaderboard), verifies the platform behaves correctly including that emails are actually delivered via Datamailer, then tears down all test data so dev stays clean.

Driven by Playwright. The test actor is an admin user that logs in through the Django admin / login form (so we avoid the Google OAuth flow that normal students use).

Why

• Catch regressions that unit tests miss (real DB, real templates, real Datamailer integration, real routing/CSRF/auth).
• Confirm a deploy is actually healthy before we trust it.
• Exercise the Datamailer path for real — today nothing verifies an email was sent and received.

Approach

• Environment: runs against https://dev.courses.datatalks.club.
• Auth: dedicated admin account, login via the admin/login form (no OAuth). Credentials from CI secrets.
• Provisioning vs. UI: use the REST API (token in env) for fast setup/teardown and assertions; use the browser (Playwright) for the user-facing flows we actually want to validate (submission forms, confirmation pages, dashboards).
• Isolation: every run creates resources under a unique, recognizable namespace (e.g. course slug e2e-smoke-<timestamp>), and a test student email at the Datamailer mock address (see sub-task). This makes parallel/retried runs safe and teardown unambiguous.
• Idempotency / teardown: teardown deletes everything created by the run; a pre-run sweep removes any stale e2e-smoke-* leftovers from a previous failed run.

Scenarios to cover

1. Availability & auth

• GET /api/health/ returns {status: ok} and the expected deployed version.
• Admin login page loads; admin can authenticate via the login form.
• Unauthenticated access to a protected page redirects to login.

2. Course & content provisioning

• Create a course (API).
• Create a homework with multiple question types (free-form, checkboxes, multiple-choice).
• Create a project (submission + peer-review config).

3. Enrollment & identity

• A test student (pre-created admin-side, email = Datamailer mock address) is enrolled in the course.
• Student profile / settings page renders.

4. Homework flow

• Submit homework answers through the UI.
• Confirmation page renders with the submitted values.
• Submission-confirmation email is received at the mock inbox (subject, template, update link).
• Score the homework (API/admin); submission shows a score.
• Leaderboard reflects the score.

5. Project flow

• Submit a project through the UI.
• Submission-confirmation email received.
• Assign peer reviews.
• Submit peer review(s) and learning-in-public / FAQ extras.
• Score the project; verify project + peer-review + LiP + FAQ score components.
• Leaderboard + project statistics update.

6. Datamailer email verification (depends on mock-inbox sub-task)

• Homework submission confirmation email delivered & content asserted.
• Project submission confirmation email delivered & content asserted.
• (Stretch) Deadline-reminder email delivered when a reminder is triggered.
• Assert idempotency keys / no duplicate sends.

7. Dashboards & stats

• Course dashboard renders with the provisioned data (completion rate, difficulty ranking).
• Wrapped / statistics pages render where applicable.

8. Teardown

• Delete submissions, peer reviews, enrollments, project, homework, course created by the run.
• Pre-run sweep of stale e2e-smoke-* resources.
• Post-run assertion that no e2e-smoke-* data remains.

Sub-task: Datamailer mock inbox (separate, in datamailer/)

To verify "email received" without spamming real inboxes, Datamailer needs a mock/test mailbox: messages sent to a designated test address are captured and exposed via an API the e2e test can poll (list recent messages for <address>), instead of (or in addition to) real delivery. Tracked separately; this e2e work depends on it. (AWS SES inbound is the fallback if we'd rather receive real mail at a controlled domain.)

Scheduling

• Trigger the suite after the dev deploy completes (CI workflow hook on dev deploy, or a scheduled cloud agent / EventBridge task).
• Surface results (pass/fail + which scenario) and alert on failure.

Open questions

• Admin login form vs. seeding a session token — which is more robust in CI?
• Real delivery + SES inbound capture, or a pure Datamailer mock-store API? (leaning mock-store)
• Where does scheduling live — GitHub Actions post-deploy job, or infra-terraform EventBridge (consistent with the deadline-reminder setup)?

Start by reviewing the existing Django REST API, admin/login flow, deployment CI configuration, and the separate datamailer/ mock-inbox sub-task. Define the Playwright suite around the listed lifecycle scenarios, using unique e2e-smoke-* data and API setup/teardown. Done means the post-deploy suite runs against dev, verifies both email flows and health/auth behavior, cleans up successfully, and reports failures.