Template catalog: versions, preview, test send, typed context

Template catalog: versions, preview, test send, typed context

Status: pending
Tags: P1, backend, api, transactional, email, docs
Depends on: None
Blocks: — (downstream unification-plan adoption work depends on this; tracked in community-base, not in this repo)

Source: unification plan issue R1.3 (community-base docs/plan/phase-1.md). Relay is the unification plan's email and jobs service; this issue sits on the plan's external critical path and may proceed independently.

Scope

• Add immutable published versions to the transactional template catalog. Each client template keeps one editable draft; PUT /api/transactional/templates/{key} keeps writing the draft; publishing snapshots the draft into a new immutable version.
• Template format: markdown body with YAML frontmatter (subject required; required_context and category optional). Rendering ports the AISL pipeline: render the context into the source with the Django template engine, convert markdown to sanitized HTML, and wrap with the shared HTML wrapper and footer (port of _render_template_with_footer and its wrapper). Plain text derives from the same source.
• Direct-HTML drafts stay supported: html_body/text_body in PUT remain a first-class content form for templates provisioned through the API (CMP provisioning uses it). Markdown and direct HTML are two content forms in one catalog; the HTML form is not a legacy path.
• Sends resolve template_key plus optional template_version (default: latest published). required_context is validated against the resolved version on every transactional send path; missing keys fail the send with a clear error before any contact, message, or queue work.
• New client-Bearer endpoints under the authenticated template scope: POST /api/transactional/templates/{key}/publish, GET /api/transactional/templates/{key}/versions, POST /api/transactional/templates/{key}/preview (returns subject, HTML, text for the draft or a version), and POST /api/transactional/templates/{key}/test-send (sends a version to an allowlisted staff address).
• Management command python manage.py import_templates --dir <path> --client <slug> creates drafts from a directory of AISL-format markdown files; the plan uses it in phase 6 and DTC uses it for its templates.
• Datamailer is pre-production: no compatibility shims, aliases, or duplicated endpoints. The AISL markdown import is an explicitly scoped client-integration requirement, built as a first-class Datamailer command and format.

Codebase notes for the implementer

• EmailTemplate (mailing/models.py) already has required_context (both plain strings and {name, description} items are accepted by normalize_required_context), example_context, default_sender_id, is_transactional, is_active. It has no version, category, or markdown-source fields; add what the design needs.
• get_transactional_template in mailing/services/transactional.py currently resolves the mutable template row for all three send paths (single send, recipient-list send, transient-list send). After this change it resolves published versions; a template with no published version must fail the send instead of silently using the draft. validate_transactional_send_payload gains template_version.
• validate_template_context (mailing/services/transactional_catalog.py) already fails sends on missing required_context with context.<name>: required (400, before contact creation). Keep that error contract; make it version-aware.
• TransactionalMessage already snapshots rendered subject/html/text at enqueue time and the sender worker sends the stored bodies, so the SQS queue payload and worker stay unchanged; persist the resolved template_version on the message for auditability.
• The frontmatter category is template metadata. It is not the send-level category_tag (subscription preference category) that sends already accept; do not conflate them.
• No markdown/YAML/frontmatter libraries are in pyproject.toml yet; add what the renderer and importer need.
• The route api/transactional/templates/<slug:template_key> accepts AISL-style keys containing underscores, so imported file stems work as keys unchanged.
• New endpoints must be registered in the in-app docs (build_openapi_spec and the endpoint map in mailing/services/api_docs.py); test_api_docs_endpoint_reference_matches_openapi_paths enforces that the endpoint reference matches OpenAPI paths.
• There is no version backfill for pre-existing templates. Sandbox environments re-run seed_demo_data and scripts/upsert_cmp_templates.py; update both to publish after upsert so existing flows keep sending.
• The staff operator UI already previews drafts at /templates/ and /templates/<id>/ (render_preview in transactional_catalog.py); keep it working and reuse service functions where sensible. It is a different surface from the new API preview.

Donor references (read first)

• docs/api.md "Transactional Email API" and the email_templates table in docs/data-model.md
• ~/git/dtc-website/_docs/specs/05-events-registration-email.md, section "Relay-owned email templates"
• ~/git/ai-shipping-labs/email_app/email_templates/ (the markdown format that must import cleanly), email_app/services/email_service.py _render_template_with_footer, and email_app/services/preview_contexts.py

Verification (from the plan issue)

• Import ~/git/ai-shipping-labs/email_app/email_templates/ into the sandbox with the command -> 50 drafts created, zero errors.
• Preview of event_registration with the AISL preview context (email_app/services/preview_contexts.py) renders without missing keys.

Acceptance Criteria

• Publish: POST /api/transactional/templates/{key}/publish snapshots the current draft as immutable version N (1-based per client and key) storing subject, the canonical content source (markdown body and/or html/text), required_context, category, and the publish timestamp. Publishing a draft whose content matches the latest published version returns that version without creating a new row.
• Immutability: no API, management command, or admin action can modify a published version; PUT and every later republish touch only the draft.
• GET /api/transactional/templates/{key}/versions lists versions with version number, subject, category, published timestamp, and which is latest; GET /api/transactional/templates/{key} returns the draft plus latest_published_version.
• Send resolution: POST /api/transactional/send, the recipient-list transactional send, and the transient-list transactional send accept optional template_version; omitted means latest published; an unknown version returns 404; a template with no published version returns 409 with a no_published_version error. A failed resolution creates no contact, no message row, and no queue payload.
• Required context: every send path validates the resolved version's required_context and rejects with 400 naming each missing context.<name> before creating the contact or message.
• Auditability: TransactionalMessage stores the resolved template_version, and the send response and dry-run response include it next to template_key.
• Preview: POST /api/transactional/templates/{key}/preview accepts {context, template_version?} (draft when no version is given) and returns {subject, html, text}; missing required context keys return a 400 listing them. Preview and send share one render code path: the same version and context produce identical subject/html/text through preview and dry_run send.
• Test send: POST /api/transactional/templates/{key}/test-send sends the given version (default: latest published) to an address matched against a deny-by-default staff-address allowlist; a non-allowlisted address is rejected with a clear error and nothing is enqueued; an allowed request goes through the normal enqueue path and marks the message as a test send in metadata.
• Import: python manage.py import_templates --dir <path> --client <slug> creates or updates one draft per *.md file (key = file stem), is idempotent on re-run, treats required_context and category as optional frontmatter, and exits non-zero with a clear message on a malformed file.
• Plan verification: importing /home/alexey/git/ai-shipping-labs/email_app/email_templates/ creates 50 drafts with zero errors, and previewing event_registration with its AISL preview context renders without missing keys.
• docs/api.md documents versions, publish, preview, test send, template_version on sends, and the frontmatter format. The in-app docs at /api-docs/ stay in sync (OpenAPI paths, endpoint reference, runnable examples), and docs/data-model.md reflects the draft/version model and transactional_messages.template_version.
• Existing flows keep working after the resolution change: seed_demo_data and scripts/upsert_cmp_templates.py publish after upsert, and the existing transactional API, dry-run, CMP template script, and LocalStack integration tests pass (updated where they now need a published version).
• [HUMAN] One test-send on the sandbox to an allowlisted staff address arrives and renders correctly in a real mail client (subject, wrapped HTML with footer, readable plain text).

Test Notes

• Unit/service tests: publish and immutability; version resolution (latest default, explicit, unknown, none published); required-context validation; frontmatter parsing (with and without optional keys); import command against a fixture directory including a malformed file; preview/send render parity.
• API/view tests: PUT draft semantics and GET shape; publish, versions list, preview, test-send (allowed, denied, another client's template); send with and without template_version; dry-run parity; api_docs/OpenAPI registration.
• LocalStack/mocked AWS tests: SES payloads for a versioned send and a test-send asserted with botocore.stub.Stubber; no test sends real email.
• UI/screenshot checks: not applicable; no visible operator UI change. Tester should state screenshots are not applicable.

Blocked by: None.

The implementation spans mailing/models.py, mailing/services/transactional.py, mailing/services/transactional_catalog.py, the transactional API routes, docs/api.md, and the seed/import paths; start by reviewing the existing LocalStack and API tests. Finish the remaining sandbox test-send to an allowlisted staff address and confirm the subject, wrapped HTML with footer, and plain text in a real mail client.