Enforce non-indexable development and preview behavior

Parent epics: #3, #9

Normative specifications:

• 02 — Development and preview behavior
• 07 — Web and application security
• 08 — Development SEO controls
• 10 — URL and SEO verification
• Open decision 16 — Analytics and tracking

Outcome

Keep https://web.dtcdev.click and every private preview demonstrably non-indexable at both the application and edge layers, while public-equivalent pages use only explicit reviewed https://datatalks.club/... canonicals. Private, authenticated, preview, error, discovery, and machine surfaces never guess a canonical or enter a shared cache.

The application, browser, runtime-source, and accepted Terraform-source implementation is complete. The current development release also passes the automated edge/browser matrix. The issue remains open only for two named, redacted, read-only live checks that cannot be inferred from edge output alone.

Current exact evidence

Closed #70 and #81 are superseded historical rollout records. They are not dependencies and their uncompleted historical A/B or pre-release OIDC ordering is not part of this issue. The current checks bind directly to the exact development release under observation.

Safety contract

Indexing and discovery

• With development noindex mode enabled, application and edge policy each set the exact single header X-Robots-Tag: noindex, nofollow on HTML, JSON, redirects, denials, errors, health, discovery files, and static responses. A downstream conflicting value is overwritten. Ordinary production public responses are not blanket-noindexed when development mode is disabled.
• GET /robots.txt returns 200, Content-Type: text/plain; charset=utf-8, and exact body User-agent: *\nDisallow: /\n; HEAD has the same status/content headers and no transfer body. There is no Allow or Sitemap directive. The development robots endpoint is absent in production mode.
• The development sitemap may expose only the production-origin route structure for compatibility testing. It remains covered by response-level noindex and the disallow-all robots.txt, contains no web.dtcdev.click location, and is never activated or submitted for indexing. An empty development sitemap is no longer required.
• A canonical is opt-in and validated as an absolute HTTPS URL on exact host datatalks.club, without userinfo, fragment, ambiguous encoding, or an unapproved query. It is supplied by the owning route/content source, never inferred from request host/path/query. Private, preview, error, redirect, robots, sitemap, JSON, and asset responses omit it unless a later owning contract explicitly requires one.

Preview, privacy, logs, and analytics

• The reusable preview guard requires an active staff session. Anonymous access uses the normal sign-in route with a safe path-only next; inactive/non-staff access is a safe 403. Authorization never uses a URL token.
• Sensitive preview query keys are rejected case-insensitively with a safe non-reflecting 400; token/query/referrer/authorization/cookie canaries do not enter response headers/body, redirects, application/compatibility/observability events, access logs, screenshots, or issue evidence.
• Every authenticated request and explicit private/PII surface, including its anonymous redirect or denial, returns private, no-store with no public or positive s-maxage. This covers Studio, Django admin/admin API, accounts/login, legacy management compatibility, learner/management/export, and preview surfaces.
• Gunicorn access output is path-only and excludes raw request target/query, client IP, Referer, User-Agent, Authorization, Cookie, and response body.
• Development rendering contains no production GTM/GA loader or identifier, makes no request to googletagmanager.com or google-analytics.com (including subdomains), and creates no _ga, _gid, _gat, or _gcl_* cookie/storage artifact.

Current edge cache boundary

The currently deployed baseline keeps all CloudFront behavior TTLs at zero while forwarding the request context Django needs and retaining application Cache-Control authority. #109 owns any later positive caching for explicitly classified anonymous public routes. If #109 lands first, it must update the source/deployed cache assertions to its accepted generated public/private matrix without weakening any noindex, private/no-store, canonical, analytics, origin-lock, or log-redaction invariant here.

Implemented and independently accepted criteria

• Exact development noindex response policy covers representative 200, 3xx, 400, 401, 403, 404, 405, safe 500, HTML, JSON, health, robots, sitemap, CSRF, and real WhiteNoise/static short-circuit responses; production-shaped negative coverage passes.
• Exact restrictive robots GET/HEAD behavior and production-mode absence are tested.
• The valid production-origin sitemap index, its section contracts, development-origin exclusion, noindex coverage, and production behavior are tested. No development sitemap submission is configured.
• Explicit-only canonical validation and rendering are tested across mapped/query-variant, unmapped, private/preview, error, redirect, JSON, robots, sitemap, asset, public editorial, course discovery, course/cohort, and learner surfaces.
• The fixture-only preview guard proves active-staff success, safe anonymous redirect, inactive/non-staff denial, session-only authorization, sensitive-query rejection, no reflection, no default canonical, and private/no-store/noindex on every outcome. No fixture preview is registered in the real URLconf.
• Every current authenticated/private/PII route class fails closed to private/no-store; conflicting shared-cache directives and credential-bearing early responses cannot bypass it.
• Query-safe Gunicorn/application/event logging and canary tests pass.
• Source and browser mutation tests reject the retired hostname, a development canonical, analytics loader/request/cookie, unsafe access-log atoms, wrong/non-overriding edge robots policy, detached policy, nonzero current-baseline TTL, and incomplete/ambiguous Terraform source.
• Development host/origin/forwarded-HTTPS behavior is exact: web.dtcdev.click and https://web.dtcdev.click; unrelated hosts are rejected outside the exact readiness exception.
• Focused tests, full Django/compatibility suites, lint, format, type, migration drift, Django/deployment checks, container checks, and core Playwright passed independently for the accepted feature and remain covered by the exact current full-profile green CI run.

Exact deployed evidence already satisfied

• Automatic run 31444623581 deployed the exact SHA/version/digest above and retained redacted HTTP plus desktop 1280×720/mobile 390×844 browser evidence.
• Public HTML, health JSON, Studio redirect/sign-in, admin API 401, deliberate 404, exact disallow-all robots, the production-origin sitemap index, and a hashed static asset all return their expected status/body class with exact X-Robots-Tag: noindex, nofollow through CloudFront.
• Public-equivalent home, hubs, course discovery, and sampled .html editorial details use exact production canonicals; Studio/sign-in/error/discovery surfaces do not emit a guessed development canonical.
• Studio/sign-in and anonymous admin API denial are private, no-store with no positive shared-cache directive. Repeated safe public observations are CloudFront misses with no positive Age under the current zero-TTL baseline.
• Browser execution at both required viewports records no GTM/GA request, analytics cookie/storage key, or service worker. Retained home, courses, Studio sign-in, and safe-404 screenshots were inspected and show the expected readable responsive pages rather than a debug/edge error.

[HUMAN] current bounded close gate

Owner: @alexeygrigorev, or one delegated development-infrastructure operator named in this issue before execution.

• [HUMAN] Applied CloudFront readback. Against the exact release being accepted, record only redacted, non-secret facts proving: web.dtcdev.click is the sole viewer alias; every active behavior attaches an overriding exact X-Robots-Tag: noindex, nofollow response-header policy; the cache/origin-request policy matches the code-owned contract for that exact SHA (all TTLs zero for the current baseline, or the accepted generated #109 matrix if #109 has deployed); application Cache-Control remains authoritative; and repeated safe edge requests have the expected cache result. Do not read, obtain, reconstruct, print, or transmit the origin-verification value or raw configuration body.
• [HUMAN] Protected application evidence. Through a reviewed in-task or otherwise protected internal method, bind evidence to the exact running SHA/version/image digest and prove Django itself returns exact noindex on representative public HTML, health JSON, private redirect/denial, deliberate error, and static responses, plus private/no-store on the private cases. Compare the redacted header/status facts with the already-passing edge evidence. Do not make the origin public, use the default CloudFront hostname, pass an origin-verification value in a command, use query/token canaries, or capture cookies, authorization, body content, PII, secrets, provider payloads, or raw request targets.

Any mismatch, unsafe access method, missing exact identity, absent header, development-origin sitemap/canonical, cacheable private response, or unverifiable application result leaves the relevant box unchecked and returns to a bounded corrective issue. If both checks pass, the named owner may check them and #36 can close without re-running retired #70/#81 history.

Non-goals

• No AWS/Terraform/GitHub configuration mutation, deployment, workflow rerun, secret access, origin bypass, Search Console action, production sitemap submission, production analytics change, or production SEO-parity claim.
• No real candidate-content preview, activation, rollback, or preview-link feature; the content-owning issue consumes the accepted guard.
• No cache optimization, WAF, country suggestion, or paid edge plan; #109 owns those changes and their separate live gate.
• No broad URL redesign or canonical invention. Production cutover remains blocked by the full #3 compatibility gate.

Dependencies and successor ownership

• Satisfied implementation prerequisites: #1, #35, and #37.
• #70 and #81 are closed superseded history and are not blockers.
• #94 must preserve the exact behavior through its Terraform/state/OIDC identifier migration; it is not a prerequisite for this close gate.
• #109 may replace the current all-zero-TTL baseline only through its generated safe public/private matrix; it is not a prerequisite for proving the current baseline.
• #76 owns recurring release-critical URL/SEO/security/browser/fault coverage after this one-time close gate.
• #3 remains the production URL/link/SEO parity epic. Closing #36 makes no production-cutover claim.

Retain the human label and leave the issue open until both current bounded checks pass. No new application commit is required when the redacted evidence matches the accepted contract.

Start with the exact deployed source, version, image digest, and CI run recorded in the issue. The remaining work is the named CloudFront readback and protected Django evidence check; done means both bounded gates are recorded with redacted facts tied to that release, without exposing secrets, raw configuration, tokens, cookies, or private request data.