Integrate Relay for transactional email and retire Datamailer

Parent epic: #6

Implementation slices: #48 (Relay-owned templates and Studio/admin proxy), #49 (durable delivery intent and Relay lifecycle), #50 (purpose wiring and Datamailer migration/cutover).

Owner decision and normative authority

The product owner selected the sibling DataTalksClub/relay service as the sole template-rendering and transactional-email delivery service for the website. New website code must call neither Amazon SES nor Datamailer directly. Datamailer is migration/reconciliation-only and receives no new website sends.

This resolved decision replaces the direct-SES/long-term-Datamailer wording currently present in the authoritative specifications. The implementation must first reconcile the specification index, open decision 10, platform architecture, events and email, Studio/admin API, security/privacy/operations, AWS sandbox, rollout, and verification to this issue. Until that documentation change lands, this recorded owner decision governs the integration.

Development course/CMP mail uses Relay sender ID courses, mapped to DataTalks.Club Courses <courses@dtcdev.click> through the existing dtcdev.click identity. No production Relay deployment, production sender/domain, or broad recipient access is approved here.

Outcome

The website commits each business mutation and one durable logical email intent atomically, then a durable website job calls Relay only after commit. Relay owns canonical versioned templates, rendering, sender policy, provider submission, queues/workers, delivery attempts/events, suppression, callbacks, reconciliation, and authoritative transport status. The website stores only the business intent and a redacted status projection needed for product behavior and operations.

A replay, worker crash, response loss, delayed callback, or cutover retry must converge without an unreviewed duplicate. Provider acceptance is distinct from delivery. An uncertain provider acknowledgement becomes ambiguous, is never blindly retried, and is resolved by reconciliation or an audited operator action. An audited manual resend is a new logical delivery with a new idempotency key linked to the original.

Audited baseline and blocking contract gaps

Read-only audit of Relay commit c0fdfac632f057f1e6a467111613928c4899b13b found client-scoped Bearer authentication, mutable GET/PUT /api/transactional/templates/{key}, POST /api/transactional/send, zero-send dry-run, sender ID mapping, GET /api/transactional/messages/{message_id}, SES event ingestion, suppression, and a durable CMP callback outbox.

The current public contract is not yet sufficient for website cutover:

• template content is mutable and has no immutable published-version contract;
• send idempotency is optional, returns an existing row without comparing a delivery-affecting request hash, and does not recover the concurrent first-insert race;
• documented dry-run is not zero-write because rendering currently upserts the contact;
• template rendering disables auto-escaping and context validation does not enforce the required safe typed/versioned contract;
• sender state is only queued/sending/sent/skipped/failed, with no expiring lease, explicit provider-accepted/delivered distinction, or ambiguous acknowledgement recovery;
• suppression is checked before queueing, not atomically again at claim immediately before provider submission;
• transport callbacks are CMP-specific Bearer callbacks containing full email/metadata, rather than a redacted generic tenant HMAC contract;
• API keys are hashed and revocable but lack per-key endpoint scopes and explicit expiry/overlap rotation.

Website audit found email_app intentionally empty while the copied course platform still owns Datamailer settings, clients, templates, callbacks, send audits, a retrying outbox, management commands, cadmin screens, and compatibility API routes. Some legacy paths can dispatch immediately, and the legacy processing state has no expiring lease recovery. These paths are migration input, not the new boundary.

Ownership and service boundaries

Website owns

• the business transaction and one unique logical EmailDelivery intent/idempotency key;
• purpose and business references, recipient/reference data allowed by retention policy, immutable Relay template key/version, minimal context snapshot/hash, and correlation IDs;
• creation of the delivery intent and durable job in the same database transaction; rollback creates neither;
• an email_app command/query service boundary shared by public views, Studio, admin API, jobs, and tests;
• after-commit Relay submission by a leased/fenced durable job carrying scalar IDs, never a network request from a request transaction, model save(), signal, or disposable callback;
• a redacted local projection of Relay status, callback/reconciliation freshness, safe reason codes, and audited operator commands.

Domain apps create intents through email_app application services and do not import worker tasks. jobs owns scheduling, wakeups, leases, retries, heartbeat, and diagnostics; it passes identifiers to email_app. Studio and /api/v1/admin/ are presentation adapters over the same services and permissions.

Relay owns

• canonical template keys, one editable draft, immutable published versions, safe validation/rendering, bootstrap content, sender resolution, and the rendered subject/plain/HTML snapshot used for a queued message;
• provider credentials/identities, submission, queues/workers, leases/attempts/backoff, SES event ingestion, suppression, provider diagnostics, generic callbacks, reconciliation, and authoritative transport state;
• the tenant-scoped versioned public API/OpenAPI contract.

The website must not create a second mutable template store, renderer, SES adapter, provider-event stack, or sender worker. A local redacted metadata cache may support an unavailable read-only view, but cannot be edited, rendered, published, or sent from.

Required Relay contract before real send

Credentials and secrets

• Provision separate tenant credentials for website runtime (transactional:send, transactional:read, templates:read) and template management (templates:read, templates:write, transactional:dry_run). Deny every undeclared endpoint and cross-tenant identifier.
• Store only credential hashes in Relay; keys expire, support overlap rotation, and are immediately revocable. Use a separate per-client callback signing secret.
• Put website credentials and callback secret in the approved secrets store through the infrastructure repository. Never place values in source, migrations, issue text, screenshots, API responses, ordinary logs, audit metadata, Terraform state/plan output, or test fixtures.
• Alert before expiry and test rotation/revocation without disabling the durable intent queue.

Versioned templates and rendering

• GET /api/transactional/templates/{key} returns redacted catalog metadata, draft revision, and the current published version reference.
• PUT /api/transactional/templates/{key}/draft requires expected_revision; stale writes return 409 without mutation.
• POST /api/transactional/templates/{key}/publish requires the draft revision plus idempotency and creates a monotonically numbered immutable version. Republishing an older version for rollback creates a new version; it never mutates history.
• GET /api/transactional/templates/{key}/versions/{version} returns that immutable version.
• POST /api/transactional/templates/{key}/render renders a named draft revision or published version with no contact/message/event/job/provider side effect. Preview, dry-run, test send, and real send use the same validator and renderer.
• Retire or deny the existing mutable published-content PUT /api/transactional/templates/{key} for the website credential.
• Escape variables by default, sanitize explicitly allowed authored rich content, reject arbitrary template tags/filters and header injection, validate missing/extra/wrong-type context, and require meaningful accessible plain/HTML parity without tracking pixels.

Every real/test send names both template_key and immutable template_version. Relay snapshots the version, resolved sender, recipient, subject, plain body, HTML body, and all delivery-affecting options before queueing so later edits or rollback cannot change queued output.

Idempotent submission and authoritative state

• POST /api/transactional/send requires a stable caller idempotency_key and computes a canonical SHA-256 request hash over tenant plus every delivery-affecting field: normalized recipient, purpose, template key/version, context, sender/reply-to, CC/BCC, headers, and message parts/options.
• Exact tenant/key/hash replay returns the original message and identifies the replay. The same tenant/key with a different hash returns 409 idempotency_conflict and creates/enqueues nothing. Concurrent first submissions converge through the database uniqueness constraint; the loser reloads and compares the winner.
• Relay exposes guarded queued, leased, provider_accepted, delivered, retryable, ambiguous, suppressed, dead, hard_bounced, and complained states. Accepted is never presented as delivered.
• Claims have owner/token/expiry and recorded attempts. An expired pre-provider lease can be reclaimed. Uncertain failure during/after the provider request becomes ambiguous and cannot auto-retry. Backoff is bounded and exhausted/permanent failures become operator-visible dead.
• Hard-bounce/complaint and applicable transactional suppression plus sender eligibility are rechecked atomically at claim immediately before provider submission.
• GET /api/transactional/messages/{message_id} returns the authoritative redacted state and ordered transition timeline. Replaying the exact send request recovers a message ID lost after Relay committed its response.

Callbacks and reconciliation

• Relay emits a durable generic callback for every transport transition needed by the website projection. The versioned payload contains stable event ID, event type/status, occurrence time, Relay message ID, website correlation/idempotency key, template key/version, and safe reason code—never body, credential, full provider payload, or full recipient.
• Relay signs <timestamp>.<raw-body> using HMAC and sends X-Relay-Timestamp plus X-Relay-Signature. The website endpoint /api/integrations/relay/events/ enforces tenant association and a five-minute replay window, deduplicates event IDs, and applies reordered events through guarded monotonic transitions.
• Relay persists callback work after its state commit and retries with bounded backoff. The website reconciles due nonterminal projections at least every five minutes in bounded batches and rechecks recently terminal deliveries daily for seven days so a missed late bounce/complaint converges. One-delivery manual reconciliation uses the same service.
• Valid unmatched provider events are deduplicated, retained redacted for seven days, retried for correlation, and become operator-visible if unresolved.

Studio and admin API implications

• Relay-backed template management lives under /studio/email/templates/; delivery diagnosis/actions live under /studio/email/deliveries/. Equivalent management operations live under /api/v1/admin/email/templates/ and /api/v1/admin/email/deliveries/.
• Template create/edit/preview/publish/republish/test and delivery inspect/reconcile/safe-retry/ambiguous-resolution/manual-resend go through the same registered website services, permissions, revision/idempotency rules, confirmations, and audit events.
• Template publication/test send and retry/resolution/manual resend are high-risk actions. Enforce the #32/#33 capability and reauthentication rules; manual resend requires fresh revision, explicit confirmation, reason, and a new linked intent/key.
• Relay timeout/unavailability/conflict produces a safe actionable UI/API error. There is no local fallback rendering or direct send.
• Mask recipient/provider identifiers by default; never expose body, context secrets/tokens, authorization material, or raw provider payload. Authenticated/PII responses remain private, no-store.
• Existing cadmin/datamailer/, /api/datamailer/events, and /api/datamailer/send-audits remain read-only migration/history surfaces only until their replacement/removal gate. They must not submit, requeue, or configure a new Datamailer send.

Datamailer migration, cutover, and retirement

1. Inventory every Datamailer caller, setting/secret reference, template/list, outbox row/state, worker/command, callback, send audit, external/provider ID, idempotency key, Studio/admin/API route, and retention requirement.
2. Import immutable history and mapping/checksum records with all outbound network, queueing, callback, immediate-dispatch, and provider submission paths disabled. Preserve traceability without copying unnecessary rendered bodies or PII. Import is repeatable and reports counts, checksums, duplicates, missing/unknown IDs, and exceptions.
3. Classify every legacy queued/in-flight item as terminal history, safe to drain in Datamailer before cutover, explicitly cancelled, or imported as send-disabled pending work requiring reviewed requeue through Relay. Import itself never sends.
4. Prove the Relay sandbox contract, scopes, allowlist/simulation, callbacks, reconciliation, and one controlled courses canary. Then freeze new Datamailer intake, drain/classify outstanding work, stop legacy sender workers, and assert one active sender per purpose before enabling that purpose in Relay.
5. Make any remaining Datamailer adapter read-only for history/reconciliation with an owner, telemetry, removal date/gate, and no submit capability. Remove credentials and runtime access only after retention, evidence, observation, and rollback gates pass; do not delete legacy records/infrastructure prematurely.
6. After one-way cutover, rollback holds new website intents and reconciles Relay. It never silently re-enables Datamailer, changes a stable idempotency request, or operates dual senders.

Dependencies and sequencing

• #31 supplies the shared durable-job/idempotency/revision/audit primitives; use them rather than a second job framework.
• Relay contract hardening above must be deployed to the sandbox and pinned by OpenAPI/consumer contract tests before website real-send is enabled.
• #48 implements the Relay-owned template contract and Studio/admin proxy; #49 implements the website intent/projection and Relay delivery hardening; #50 wires approved purposes and performs migration/cutover. This issue closes only when all three slices and the cross-repository evidence pass.
• #32 and #33 are required before privileged Studio/admin API credential, publication, retry/resolution, or manual-resend operations are enabled.
• #9 and the infrastructure repository own sandbox resources, secret containers/policies, sender/provider configuration, and alarms. This issue authorizes no direct AWS mutation or production rollout.
• #22 is the only remaining product choice: each non-course purpose still needs its approved owner, audience, sender/reply-to, template/context, idempotency/version inputs, and retention class. The approved development courses path may proceed; every unapproved purpose/sender fails closed.
• #106 is a superseded duplicate of this existing issue and is not a separate implementation dependency.

Acceptance criteria

• Normative email/architecture/security/infra/rollout/verification specs record Relay ownership, prohibit website-direct SES and new Datamailer sends, and preserve after-commit durable-job semantics.
• Relay sandbox identifies the exact deployed commit/OpenAPI version and passes pinned website consumer contract tests before real send is enabled.
• Scoped expiring runtime/template credentials and a separate callback secret are provisioned; rotation, revocation, wrong-scope, and cross-tenant denial pass without secret leakage.
• Relay drafts/revisions and immutable published versions pass concurrent edit, idempotent publish, rollback-as-new-version, safe render, typed context, escaping/sanitization, plain/HTML parity, and queued-snapshot tests.
• Mandatory request-hash idempotency returns the original for exact replay, returns 409 for changed work, and converges concurrent first submissions to one Relay message and one provider submission.
• Business state, one website EmailDelivery, and one durable job commit atomically; rollback creates none and no network side effect occurs before commit.
• Job replay, response loss, lease expiry, retry exhaustion, and before/during/after-provider crashes obey guarded state, bounded retry, accepted-versus-delivered, and ambiguity semantics without blind resend.
• Claim-time suppression prevents a send when a hard bounce/complaint arrives after queueing.
• Generic callbacks are HMAC/timestamp authenticated, tenant-scoped, redacted, deduplicated, reorder-tolerant, durably retried, and recoverable through scheduled/manual reconciliation.
• Website and Relay logs, audit, jobs, UI/API, callbacks, metrics, screenshots, and test artifacts contain no secrets, bodies, raw provider payloads, or unnecessary/full recipient data.
• Studio/admin API template and delivery capabilities have permission, revision, idempotency, confirmation, audit, error, and side-effect parity; Relay-unavailable behavior has no local fallback.
• Only the approved development courses sender and allowlisted/simulated recipients can send; unknown purposes/senders and production configuration fail closed.
• Datamailer import is repeatable and send-disabled; every outstanding item is classified; freeze/drain evidence proves zero unclassified in-flight work and one active sender per purpose.
• After cutover, no website runtime path can submit to Datamailer or SES; rollback holds/reconciles rather than dual-sending, and legacy removal follows retention/observation gates.
• Focused unit, PostgreSQL concurrency, integration/contract, failure-injection, migration, security/redaction, and Studio/admin API/browser tests pass.

Focused validation scenarios

1. Business transaction rollback, successful commit, trigger replay, concurrent submit, timeout before Relay commit, and response loss after Relay commit converge to one website intent and one Relay message.
2. Exact replay versus changed recipient/template version/context/sender/options proves original response versus 409, including a concurrent insert race.
3. Crash before claim, after claim/before provider, during uncertain provider acknowledgement, after provider acceptance, and after callback enqueue proves lease recovery or explicit ambiguity without automatic duplicate send.
4. A complaint arrives after queueing but before claim; claim suppresses. Duplicate/out-of-order/missing/invalid-signature callbacks and a late bounce converge through reconciliation without status regression.
5. Edit/publish/rollback while a message is queued; its template version, sender, subject, plain body, and HTML body remain byte-for-byte unchanged. Malicious context/source, unsupported template constructs, unsafe URLs, and header injection fail safely.
6. Dry-run/preview creates no contact, message, event, job, queue item, callback, or provider call. Expired/revoked/wrong-scope/cross-tenant credentials fail without object-existence leakage.
7. Import duplicate/missing/unknown Datamailer IDs while sending is disabled; repeat import produces the same mappings/counts. Freeze/drain/cutover rehearsal proves no new Datamailer intake and no dual sender.

Browser and API parity

At desktop and mobile widths, an authorized operator creates/edits/previews/publishes/republishes/tests a Relay template, sees validation and stale-revision errors, and inspects redacted queued/accepted/delivered/retryable/ambiguous/suppressed/dead/bounced/complained delivery states. Exercise reconcile, safe retry, ambiguous resolution, and manual resend with confirmations; verify negative roles, stale/double-submit protection, Relay-unavailable/lagging states, masked PII, keyboard/focus/error behavior, and equivalent admin API results. Store and inspect screenshots only below .tmp/screenshots/.

Safe sandbox/live verification

• Automated tests use fakes, captured requests, console/in-memory behavior, Relay dry-run, or SES simulator addresses and never send external email.
• Before any live sandbox canary, verify the deployed Relay commit/OpenAPI, credential scopes, recipient allowlist, sender ID courses, callback endpoint, reconciliation scheduler, alarms, and an empty/classified Datamailer in-flight queue.
• Run exactly one explicitly approved allowlisted development recipient through the normal committed-intent/durable-job path. Capture only redacted IDs/timestamps/status evidence showing queued/provider-accepted/delivered convergence; never capture the address, body, token, credential, or provider payload.
• Fault Relay after the website commit and during callback delivery, then prove backlog/alerts/reconciliation recover without Datamailer fallback. Production remains disabled.

Non-goals

• Marketing/newsletter campaigns, marketing-consent inference, tracking pixels, or engagement analytics.
• A website-owned canonical template-content store, renderer, SES adapter, provider queue/worker, or provider-event stack.
• New Datamailer sends, dual-send fallback, or deletion of legacy data before migration/retention/rollback gates.
• Relay production deployment, production DNS/sender/reply-to approval, or broad infrastructure mutation.
• Treating provider acceptance as delivery, automatically retrying ambiguity, or exposing secrets/full PII/message bodies/raw provider payloads.

Start with implementation slices #48, #49, and #50, then reconcile the specifications listed under _docs/specs/. Review the existing email_app, jobs, and copied course-platform paths described in the audit, along with /api/integrations/relay/events/. Done means the Relay contract, durable intent flow, migration, and verification requirements are implemented without new direct SES or Datamailer sends.