Plan monitoring and observability for course management platform

Goal

Add practical monitoring and observability for the course management platform without self-hosting Grafana, prioritizing free or low-cost hosted options.

We want to know what is happening around releases and catch application health problems early:

• unhandled errors and exceptions
• release/deploy health and regressions
• important student/staff events
• page/API availability
• submission/scoring/email pipeline health

Current app inventory

The app is a Django project deployed to AWS ECS from GitHub Actions:

• GitHub Actions builds Docker images, pushes to ECR, and updates ECS task definitions in .github/workflows/deploy-dev.yaml, .github/workflows/deploy-prod.yaml, and deploy/deploy_*.sh.
• deploy/update_task_def.py writes the Docker tag into the ECS VERSION environment variable.
• /api/health/ returns {"status": "ok", "version": settings.VERSION} from api/views/health.py.
• Django already emits JSON logs to stdout in non-local environments through course_management/settings.py.
• No Sentry, OpenTelemetry, AWS X-Ray, Datadog, New Relic, Honeybadger, or hosted Grafana dependency was found in pyproject.toml.
• Datamailer has useful domain-level health data already stored in data.models: DatamailerOutboxEvent, DatamailerOutboxDispatchRun, DatamailerContactEvent, and DatamailerSendAudit.
• There are management commands that already summarize Datamailer health: datamailer_outbox_status, datamailer_send_status, and datamailer_callback_status.

Important events to monitor

Authentication and identity

• Login page views: accounts/views/login.py.
• Successful/failed OAuth login and account linking: accounts/auth.py via ConsolidatingSocialAccountAdapter.pre_social_login.
• New users created from social login: ConsolidatingSocialAccountAdapter._create_user_for_email.
• Multiple existing users found for the same social email: currently logged as warning in accounts/auth.py.
• Token-auth API failures: accounts/auth.py::token_required.
• Staff impersonation start/stop: django-loginas under /admin/ and accounts/views/impersonation.py::stop_impersonating.

Privacy note: current OAuth logging includes full provider response JSON. Before expanding log retention, review this for PII/token exposure and either remove it or redact it.

Page/API traffic and availability

• Public course list/course/detail pages from courses/urls.py.
• Student pages: course dashboard, homework, project, leaderboard, enrollment.
• Staff admin pages under cadmin/urls.py.
• API requests under api/urls.py, especially staff-token mutation endpoints.
• /api/health/ uptime and version checks.
• 4xx/5xx counts, request latency, and response sizes from ALB/ECS/gunicorn access logs if available.

Student lifecycle events

• Registration campaign submitted: courses/views/registration.py::registration_campaign_view.
• Enrollment created/updated/toggles changed: courses/views/course_enrollment.py.
• Account preference/toggle/timezone/email-preference changes in accounts/views/*.
• Leaderboard complaint submitted/resolved: courses.models.course.LeaderboardComplaint and cadmin/views/enrollment.py.

Homework events

• Homework viewed and submitted: courses/views/homework.py and courses/views/homework_submission.py.
• Submission update versus first submission: homework_submission_for_user.
• Homework validation errors: courses/views/homework_post_preview.py.
• Homework scoring started/completed/failed: courses/scoring.py and API/staff routes.
• Staff edits to submissions and correct answers: cadmin/views/homework.py, cadmin/views/homework_submission_edit.py.
• Score notification sends: Datamailer outbox/send audit events.

Suggested event fields: event_name, course_slug, homework_slug, homework_id, submission_id, user_id, enrollment_id, is_update, question_count, duration_ms, status, error_code.

Do not log answer text or free-form comments.

Project and peer review events

• Project viewed/submitted/deleted: courses/views/project.py and courses/views/project_submission_edit.py.
• Project validation errors.
• Peer reviews assigned: courses/project_assignment.py.
• Peer review submitted: courses/views/project_eval_submit.py and courses/views/project_eval_submit_save.py.
• Optional peer review add/delete: courses/views/project_eval_actions.py.
• Project votes: courses/views/project_eval_submit.py::project_eval_vote_response.
• Project scoring started/completed/failed: courses/project_scoring.py and API/staff routes.
• Staff submission edits: cadmin/views/project_submission_edit.py.

Suggested event fields: event_name, course_slug, project_slug, project_id, submission_id, review_id, reviewer_enrollment_id, user_id, assigned_count, submitted_count, passed_count, duration_ms, status, error_code.

Do not log repository contents, notes to peer, or free-form comments. GitHub repo URLs and commit IDs are probably acceptable as structured fields, but avoid over-indexing them in metric dimensions.

Staff/API mutations

• Course/homework/project create/update/delete through api/views/*.
• Homework question create/update/delete through api/views/questions.py.
• Registration campaign create/update through api/views/registration_campaigns.py.
• Certificate/graduates imports and bulk certificate update endpoints.
• Staff-triggered homework/project scoring and review assignment.
• Unauthorized/invalid token attempts.

Suggested event fields: event_name, actor_user_id, course_slug, object_type, object_id, method, path_name, status, error_code.

Datamailer and background-like work

• Outbox enqueue/dispatch/ack/retry/fail/dead from course_management/datamailer_outbox*.py.
• Outbox run success/failure from DatamailerOutboxDispatchRun.
• Send audit success/failure from DatamailerSendAudit.
• Webhook callback accepted/duplicate/rejected from api/views/webhooks.py and validation helpers.
• Registration confirmation, homework/project confirmation, score notifications, certificates, peer-review notifications.
• Scheduled process_datamailer_outbox execution should have heartbeat/check-in monitoring.

Recommended monitoring architecture

Phase 1: cheapest useful baseline

Use CloudWatch for infra/log retention plus Sentry or Honeybadger for application errors and release visibility.

1. Keep structured JSON stdout logs and ship ECS task logs to CloudWatch Logs.
2. Add log fields consistently: event, release, environment, request_id, user_id, course_slug, object_type, object_id, status.
3. Add Sentry SDK or Honeybadger for Django exceptions.
4. Set release=settings.VERSION and environment=dev|prod, matching the ECS VERSION value already set by deployment.
5. Add GitHub Actions release/deploy notification to the error tracker after deploy.
6. Add an external uptime monitor for /api/health/ and one or two key public pages.
7. Add CloudWatch alarms for ECS service health, 5xx rate, task restarts, memory/CPU, and log error patterns.
8. Add a daily/hourly scheduled health script that runs existing Datamailer status commands and emits one compact JSON log line or CloudWatch metric.

This gives immediate error tracking and release correlation without running Grafana.

Phase 2: app-level events

Add a tiny internal event helper, for example monitoring/events.py, that logs a structured event and optionally increments metrics later.

Initial events:

• auth.login_success, auth.login_failed, auth.user_created, auth.social_account_linked
• registration.submitted
• enrollment.created, enrollment.updated
• homework.submitted, homework.validation_failed, homework.scored
• project.submitted, project.deleted, project.peer_reviews_assigned, project.review_submitted, project.scored
• api.mutation, api.auth_failed
• datamailer.outbox_failed, datamailer.outbox_backlog, datamailer.send_failed, datamailer.callback_received

Important: do not emit high-cardinality dimensions as CloudWatch custom metrics. Keep detailed IDs in logs/error context. Use metrics only for low-cardinality counters such as event name, status, environment, and object type.

Phase 3: dashboards and alerts

Build a small number of hosted dashboards/alerts:

• Release overview: deploy version, error count since deploy, 5xx count, health check status.
• Student activity: registrations, enrollments, homework submissions, project submissions, peer reviews.
• Course operations: scoring runs, review assignment, leaderboard updates.
• Datamailer: outbox backlog, failed events, last successful run, send audit failures, callback freshness.
• API/admin: mutation counts, unauthorized requests, staff-triggered destructive actions.

Option matrix

Option	Covers	Likely low-volume cost	Pros	Cons
AWS CloudWatch Logs + alarms	ECS logs, error log patterns, 5xx metrics, CPU/memory, dashboards	Often near free if logs stay under free/low GB volumes; official pricing includes paid ingestion/storage after free usage	Already AWS-native; no new vendor; easy ECS integration	Weak error grouping; custom metrics can get expensive with high-cardinality labels
AWS X-Ray / CloudWatch Application Signals	Traces, latency, service map	Cheap with sampling at low volume; AWS example shows 148,800 sampled traces costing $0.24 recorded after free tier	Native AWS tracing; good for slow requests and DB/API latency	More setup; trace volume needs sampling discipline
Sentry hosted	Django exceptions, release health, deploy correlation, traces, uptime/cron monitors, alerting	Free Developer plan for one user and small volume; Team starts at $26/mo annually	Best fit for “what broke after release?”; Django integration is straightforward; release/deploy support	Free plan is one user and limited events; not ideal as the only event analytics store
Honeybadger hosted	Exceptions, deploy tracking, logs/performance, uptime, check-ins	Free Developer plan for one user; paid Team starts around $26/mo	Simple all-in-one app health product; includes check-ins and uptime	Smaller ecosystem than Sentry; less common if future contributors expect Sentry
PostHog	Product/app events, page views, funnels, lightweight audit trail, optional session replay/error tracking	Free tier includes 1M analytics events and 100K exceptions/month	Best fit for “what are students/staff doing?” without turning CloudWatch into an analytics store; billing limits available	Not infra monitoring; must be careful with PII and free-form submission data
Better Stack	Uptime, logs/traces, incidents, status page, web events	Free tier includes small telemetry/monitoring allowances; responder/on-call is paid	Strong hosted uptime/logs/status-page workflow; no self-hosting	Broader platform; costs depend on telemetry volume and responders
Grafana Cloud hosted	Hosted Grafana/Loki/Tempo/Prometheus-style metrics/logs/traces	Free hosted tier exists; Pro starts at $19/mo plus usage	Satisfies “no self-hosted Grafana” while still using Grafana dashboards	More instrumentation/ops concepts than Sentry/Honeybadger; watch cardinality
UptimeRobot	External uptime checks	Free plan: 50 monitors with 5-minute interval; paid Solo starts at $7/mo annually	Very fast way to monitor /api/health/ and public pages	Uptime only; no app errors or release correlation
Healthchecks.io	Cron/job heartbeats	Free for 20 checks; paid Business is $20/mo or $16/mo annually	Very cheap Datamailer outbox/scheduled job heartbeat monitoring	Does not cover exceptions, product events, or request traffic
StatusCake	External uptime, SSL/domain/page-speed checks	Free plan has 10 uptime monitors at 5-minute intervals	Simple hosted uptime alternative to UptimeRobot	Less generous free monitor count than UptimeRobot
New Relic	All-in-one APM/logs/metrics/synthetics/errors	Free tier includes 100 GB/month data ingest and one full platform user	Large free ingest allowance and broad platform	Heavier than needed; costs can jump with additional full users or ingest
Honeycomb	OpenTelemetry traces/events and high-cardinality debugging	Free tier up to 20M events/month; Pro starts at $150/mo	Strong if tracing and high-cardinality backend debugging become central	Less direct than Sentry/Honeybadger for Django exception triage; paid tier is a bigger step up
EventBridge Scheduler + ECS task/management command	Scheduled health checks/Datamailer status emitters	EventBridge Scheduler has 14M free invocations/month	Cheap way to run status commands and heartbeat checks	Need to wire output into logs/metrics and alerts

Suggested decision

Recommended starting stack:

1. Sentry hosted for Django error tracking and release/deploy health.
2. PostHog for product/audit-style events such as login, registrations, page views, submissions, peer-review actions, and staff mutations.
3. CloudWatch Logs/alarms for ECS infra, log retention, and low-cardinality app counters.
4. UptimeRobot, StatusCake, or Sentry Uptime for /api/health/ and key page checks.
5. Healthchecks.io or Sentry Cron Monitoring for periodic datamailer_*_status and process_datamailer_outbox heartbeat/check-in monitoring.

Why this combination:

• It avoids self-hosting Grafana.
• It gives release-aware error tracking quickly.
• It reuses the existing health endpoint and VERSION deployment flow.
• It keeps CloudWatch custom metrics disciplined and cheap.
• It puts high-volume user/application events into a product-event tool with a generous free tier instead of into expensive high-cardinality CloudWatch custom metrics.

If the team wants the cheapest possible single external vendor, Honeybadger is a reasonable alternative to Sentry because it combines error tracking, deploy tracking, uptime, and check-ins. If product analytics becomes the primary need, PostHog should be the second integration after error tracking. If dashboards/log analytics become the primary pain, consider Better Stack or hosted Grafana Cloud next. New Relic has a generous free ingest tier but is probably more platform than this app needs initially.

Implementation checklist

• Add an error tracker SDK (sentry-sdk[django] or Honeybadger equivalent) with DSN/env vars disabled by default locally.
• Set release/version from settings.VERSION.
• Add GitHub Actions deploy/release notification for dev/prod.
• Add request ID middleware or configure one if already present.
• Add structured event helper for important domain events.
• Instrument homework/project submission, scoring, registration, enrollment, Datamailer, API auth failure, and staff mutation paths.
• Redact or remove OAuth provider response logging before retaining logs externally.
• Add CloudWatch log retention policy and alarms.
• Add external uptime check for /api/health/ and at least one public course page.
• Add scheduled Datamailer health/check-in job.
• Document event names and privacy rules.

Sources checked

• Sentry pricing and plan limits: https://sentry.io/pricing/
• Sentry Django integration: https://docs.sentry.io/platforms/python/integrations/django/
• Sentry releases/release health: https://docs.sentry.io/product/releases/
• Sentry GitHub Actions release automation: https://docs.sentry.io/product/releases/setup/release-automation/github-actions/
• AWS CloudWatch pricing: https://aws.amazon.com/cloudwatch/pricing/
• AWS EventBridge Scheduler pricing: https://aws.amazon.com/eventbridge/pricing/
• Honeybadger plans: https://www.honeybadger.io/plans/
• Better Stack pricing: https://betterstack.com/pricing
• Grafana Cloud free tier: https://grafana.com/products/cloud/free-tier/
• UptimeRobot pricing: https://uptimerobot.com/pricing/
• PostHog pricing and error tracking pricing: https://posthog.com/pricing and https://posthog.com/docs/error-tracking/pricing
• Healthchecks.io pricing: https://healthchecks.io/pricing/
• StatusCake pricing: https://www.statuscake.com/pricing/
• New Relic pricing: https://newrelic.com/pricing
• Honeycomb pricing: https://www.honeycomb.io/pricing

Start by reviewing the current deployment and health entry points: .github/workflows/deploy-dev.yaml, .github/workflows/deploy-prod.yaml, deploy/deploy_*.sh, deploy/update_task_def.py, api/views/health.py, and course_management/settings.py. Compare the Phase 1 recommendations with the listed Datamailer commands and data models. Done should be a decided, scoped monitoring plan with an agreed hosted option, rollout phases, alerts, privacy safeguards, and implementation entry points.