Define the redacted observability and correlation contract

Parent: #66

Product outcome

Give every application domain one safe, deterministic way to emit correlatable operational events without exposing user data or letting an observability failure affect the request, transaction, durable job, callback, or worker result. This issue freezes the website-owned event envelope and propagation seam only; it does not define measurements, targets, queries, alerts, infrastructure, recovery, or operator procedures.

Normative authority

• _docs/PROCESS.md — role-separated lifecycle, versioned verification plans, backend-only browser tier, and sensitive-artifact rules.
• _docs/specs/01-platform-architecture.md — one Django deployment, domain ownership, durable after-commit work, safe degradation, and the sealed application release identity.
• _docs/specs/07-security-privacy-operations.md#observability — required request/job/message, route/operation, timing/status, actor, release/domain/queue/delivery context; exclusions; immutable release fields; bounded metrics; optional tracing failure behavior.
• _docs/specs/10-verification-strategy.md — redaction, correlation, failure, release-identity, and artifact verification.
• _docs/architecture/shared-primitives.md#execution-context-and-redaction — bounded ContextVar request/correlation/job IDs, durable-job propagation, ServiceContext, and the shared redaction boundary.
• _docs/architecture/app-boundaries.md — core owns shared context/redaction while each domain owns the meaning and correctness of its events.
• _docs/runbooks/development-release.md#current-release-identity-contract — schema-2 VERSION, full SOURCE_SHA, and immutable IMAGE_DIGEST identity.
• Closed #31 and #141 supply the accepted context, service, durable-job, request-boundary, and redaction foundations. #66 remains the parent operations epic.

Current baseline and gap

course_management.observability.events.AppEvent currently produces an informal flat schema selected by OBSERVABILITY_EVENT_SCHEMA_VERSION, appends request path/method, derives a potentially identifying distinct_id, and normalizes separately inside each backend. core.context, RequestIdMiddleware, ServiceContext, and jobs.execution already bind and reset safe request/correlation/job values. Runtime settings already expose the correct release triplet. The log/no-op backends and generic CloudWatch EMF counter catch backend exceptions, but there is no frozen envelope, semantic event version, canonical byte representation, safe route rule, message/delivery propagation seam, field/cardinality budget, or complete failure contract.

Scope

1. Freeze envelope schema 2

One normalization pass creates one immutable payload which every configured backend receives. The canonical object always contains the following keys; null means that safe context was genuinely unavailable and is distinct from an empty string:

Field	Type / rule	Requirement
schema_version	integer, exactly 2	required
event	registered lower-case dotted name, max 96 characters	required
event_version	integer 1..255, owned by that event's domain	required; existing named events begin at 1
occurred_at	UTC RFC3339 with exactly millisecond precision and Z	required; captured once per emission
environment	validated runtime environment label	required
version	exact runtime settings.VERSION, including the accepted local fallback	required
source_sha	exact full lower-case runtime SHA, or null only for the accepted local fallback	required
image_digest	exact immutable sha256: runtime digest, or null only for the accepted local fallback	required
request_id, correlation_id, job_id, message_id	safe opaque context IDs, each max 128, or null	keys required; values contextual
route, operation	registered safe labels, max 128, or null	keys required; at least the available transport route or application operation is populated
duration_ms, queue_age_ms	non-negative integers, or null	keys required; values contextual
status	registered lower-case outcome code, max 32, or null	key required; terminal events populate it
error_class	registered stable lower-case error code, max 64, or null	key required; failure events populate it and never exception text
actor_class	exactly one of anonymous, member, staff, service, system, unknown	required; no actor/user/account ID
content_release, course, cohort, event_ref, delivery_ref	safe opaque domain references, max 128, or null	keys required; values supplied only by the owning domain
attributes	bounded redacted object described below	required; {} when unused

distinct_id is not part of schema 2. No user-derived identifier substitutes for it. The legacy record_event(name, request=..., user=..., distinct_id=..., properties=...) call shape remains source-compatible for one migration release: name maps to event, the default semantic version is 1, safe non-reserved properties map into attributes, and user maps only to actor_class. A supplied distinct_id is discarded after redaction validation and never serialized or sent to a backend. New code uses the explicit schema-2 builder/emitter and cannot override envelope or release-identity fields through attributes.

Schema 2 is a code-owned constant, not an environment-selectable dialect. Startup/settings validation rejects an incompatible configured schema rather than silently producing a mixed shape. Existing schema-1 log records remain historical evidence; this change does not rewrite or dual-emit them. A backward-compatibility test freezes the one-release legacy-call adapter. Any future envelope change increments schema_version; any semantic meaning/required-domain-field change increments that domain event's event_version. Additive optional attributes alone do not change either version.

2. Deterministic redaction and canonical serialization

Normalize and redact before any backend sees the event. Canonical JSON is UTF-8, has lexicographically sorted object keys at every depth, preserves list order, uses compact separators, emits JSON booleans/null and base-10 finite integers only, escapes according to one fixed JSON encoder, has no NaN/Infinity and no trailing newline. The same normalized object and canonical bytes are used for every backend; a golden fixture freezes the exact bytes.

attributes accepts at most 32 string keys, depth 4, 128 total nodes, 32 list items, 256 characters per key or string value, and a 16 KiB maximum canonical event. Unknown objects, bytes, sets, cycles, non-string keys, non-finite numbers, excess depth/items/nodes/length, and an over-budget event fail event validation. They do not invoke str() and do not reach a backend.

Sensitive-key entries are omitted. Sensitive or canary-bearing values that survive an innocuous key are replaced with the shared deterministic redaction marker before size/canonical checks. The canonical payload must never contain raw email, profile/submission/question/content, Cookie, Authorization, session/CSRF value, complete query, raw IP, country/origin-verification header, preview/management/Slack link, request/response body, credential, secret, provider payload/response, database URL/row/export, exception message, traceback text, or user/account identifier.

route is the resolved Django route name or a registered route template, never request.path, path_info, get_full_path(), query text, or a URL containing viewer-controlled segments. operation, status/error codes, and domain references pass explicit validators and the shared sensitive-text guard. Delivery context uses the website's safe logical reference; raw provider message IDs/payloads are not accepted.

3. Cardinality contract

The generic EMF counter remains behaviorally compatible and keeps exactly environment and event as dimensions in this issue. The envelope additionally classifies only event_version, status, error_class, and actor_class as eligible bounded labels for a later metric contract. No backend may turn request/correlation/job/message IDs, route, operation, release identity, content/course/cohort/event/delivery references, duration/age, or arbitrary attributes into metric dimensions.

Registered event/route/operation/status/error labels are finite code-owned sets in their owning module. Runtime/user/provider input cannot create a new label. This issue adds validation and tests, not target values, aggregates, queries, thresholds, metrics, alarms, or dashboards.

4. Context propagation boundaries

• HTTP middleware validates or replaces external request/correlation headers, binds both values for the request, returns them in response headers, and resets them in finally. Observability derives route only after safe Django route resolution.
• Application services snapshot the current request/correlation context through ServiceContext; no request object, header collection, body, user object, or credentials cross the service boundary.
• Durable enqueue persists only the already accepted request/correlation scalars. Worker execution rebinds them plus the stable durable job_id for exactly one attempt and resets all tokens in finally, including nested/error paths.
• Delivery work binds a safe website logical message_id/delivery_ref explicitly for one handler invocation. An authenticated callback or reconciliation starts with its own request/correlation context, resolves the website logical reference through the owning domain, and only then binds it. Provider input is never trusted as context.
• Content, courses/cohorts, and events pass only their safe opaque reference through the common builder; they retain ownership of event names, semantic versions, statuses, and attribute validation. Common observability code does not query domain models or infer domain meaning.
• Context never propagates implicitly across processes, threads, callbacks, or retries. Only accepted persisted scalars cross an asynchronous boundary, and nested scopes restore the exact enclosing context.

Provide domain-owned schema/validator fixtures that prove the common seam for request, durable-job, delivery, content-release, course/cohort, and event context. Migrating or inventing every domain's event catalog is not part of this issue; later domain work owns its semantic events against this accepted seam.

5. Failure behavior

Event creation/validation, canonicalization, and every backend invocation are outside business transactions and must never change the request response, committed mutation, durable-job state, callback acknowledgement, retry classification, or worker exit result. An invalid event is dropped before all backends and produces one bounded safe diagnostic containing only a stable rejection code and a validated/placeholder event label.

Configured backends run independently in declared order. One backend failure does not prevent later backends. The failure diagnostic contains only backend class/slot and a stable failure code; it never includes exception text, traceback, backend payload, credentials, or the rejected canonical event. Failure diagnostics do not recurse through the failing event backend. An unknown/misconfigured backend fails closed at process boot outside local/test; local/test may explicitly use noop.

6. Ownership and integration

Move/locate the common contract under the shared website boundary (core or a dependency-neutral shared observability module) so accounts, courses, events, content_sync, email_app, and jobs can consume it without importing course_management internals or creating reverse imports. Preserve a compatibility re-export for existing callers during the one-release migration window.

Each owning domain defines and tests its own event name/version, status/error allowlists, safe reference construction, and semantic attributes. Shared code owns only envelope validation, context capture, redaction, canonical serialization, backend isolation, label budgets, and release identity. No model or migration is required merely to store observability events.

Dependencies and sequencing

• Accepted foundations: closed #31 and #141.
• Parent/reference only: #66. The parent remaining open does not block this bounded child.
• No open issue is an implementation prerequisite. In particular, #49 need not be complete because this issue freezes only the safe logical delivery seam and does not define Relay/delivery behavior.
• #266's service-target/query catalog must consume the accepted schema and therefore follows this issue; it is not a prerequisite.
• This lane is independent of the provider-neutral backup receipt and may proceed in its own worktree.

Acceptance criteria

• Schema 2 and semantic event version 1 are code-owned, validated, documented, and emitted with every required key/type/null rule above; local and deployed release identities match the runtime contract exactly.
• A one-release compatibility adapter accepts existing record_event calls without serializing distinct_id, raw paths, reserved-field overrides, or user identity; schema-1 history is neither rewritten nor dual-emitted.
• One normalization pass produces byte-identical canonical JSON for every backend; golden tests cover ordering, encoding, UTC millisecond time, list order, nulls, numeric rules, and the 16 KiB boundary.
• Attribute shape, string, node, list, depth, label, ID, and payload bounds reject malformed/oversized input before any backend without calling str() on unknown values.
• Sensitive-key and canary fixtures prove all prohibited values are absent from normalized objects, canonical bytes, log records, EMF payloads, diagnostics, test artifacts, and failure output.
• Request route capture uses a resolved name/template and never raw path/query; actor context is classification-only; release fields and reserved envelope keys cannot be overridden.
• Request/correlation/job/message context propagates and resets exactly across request, service, durable enqueue/attempt/retry, delivery handler, callback/reconciliation, nested-scope, exception, and concurrent-task cases; missing context remains explicit null.
• Request, durable-job, logical-delivery, content-release, course/cohort, and event domain fixtures each validate against the common seam without a shared-module import of domain models.
• Generic EMF dimensions remain exactly environment,event; property fuzzing proves no identifier, release field, route/operation, duration/age, domain reference, or arbitrary attribute can become a dimension.
• Invalid events and throwing/misconfigured backends are isolated exactly as specified: business outcomes are unchanged, later backends still run, and diagnostics contain only stable safe codes/labels with no exception or payload text.
• Import-boundary tests prove domain-neutral ownership plus the temporary compatibility re-export; no reverse app dependency, model, migration, external call, or provider SDK is introduced.
• Focused observability, context, middleware, durable-job, redaction/security-canary, release-identity, settings/bootstrap, and representative domain conformance tests pass; migration drift and repository checks pass.
• The engineer records the exact versioned verification plan and evidence required by _docs/ci/change-selective-ci.md; the independent tester recomputes it and reports every component exactly once.

Required scenarios

1. Exact canonical golden event; reordered source dictionaries; non-ASCII and escaped text; exact size/bound boundaries; local and deployed identity.
2. Missing/invalid/credential-shaped external IDs; request to service to enqueue to worker; nested attempt and retry; logical delivery handler; callback/reconciliation with a distinct request but the resolved logical reference; concurrent contexts; all tokens reset.
3. Resolved named route versus viewer-controlled path/query/token; anonymous/member/staff/service/system actor classes; attempted reserved-field override.
4. Sensitive field aliases, nested canaries, email/URL/provider/credential/body/exception fixtures, unknown objects, cycles, sets, bytes, non-string keys, NaN/Infinity, excess containers, and payload overflow.
5. Backend A succeeds/fails, backend B still runs; invalid event reaches neither; safe diagnostic logging itself cannot recurse; unknown backend configuration fails at the defined boot boundary.
6. Metric payload with every envelope/domain field proves the exact two dimensions and absence of all prohibited dimension values.
7. Existing schema-1-style caller through the temporary adapter and a native schema-2 caller produce the expected safe schema-2 envelope without duplicate emission.

Browser, rendering, and screenshots

No template, stylesheet, route, navigation, page copy, or operator UI changes are in scope. Render impact and screenshots are not_applicable. Under _docs/PROCESS.md, the independent tester uses the backend-only Playwright smoke tier selected by the versioned plan; no unrelated product screenshot may substitute for contract evidence.

Explicit non-goals

• No service targets, target values, aggregates, measurements, queries, windows/exclusions, thresholds, SLOs, alert ownership/routing, dashboard panels, or runbooks.
• No Terraform, AWS resource/read/write, CloudWatch dashboard/alarm change, log subscription, provider configuration, deployment, or production action.
• No backup verification, snapshot protocol, restore, tombstone replay, delivery/outbox reconciliation behavior, activation control, rollback, fault/expiry drill, or recovery evidence.
• No OpenTelemetry/tracing dependency, exporter, trace/span IDs, baggage propagation, sampling policy, or runtime database toggle. Optional tracing is deferred; any later tracer must be boot-configured, consume only this normalized redacted envelope, and fail safely.
• No new domain product behavior, complete event catalog, metrics implementation, provider call, direct sender, raw provider identifier, historical resend, or inference that provider acceptance means delivery.
• No persistence of event payloads, observability database model/migration, public/admin API, browser/operator surface, or redesign.
• No PII, content, raw URL/query/path, header, body, cookie, token, secret, credential, provider payload, exception text/traceback, database data, or high-cardinality identifier in metrics, logs, traces, screenshots, or artifacts.

Start with _docs/specs/07-security-privacy-operations.md#observability, _docs/architecture/shared-primitives.md#execution-context-and-redaction, and the current course_management.observability.events.AppEvent. Trace core.context, RequestIdMiddleware, ServiceContext, and jobs.execution to understand existing propagation and reset behavior. Done means the schema-2 envelope, canonical redaction, propagation seams, domain fixtures, and failure behavior are validated without changing business outcomes.