Make SAM artifacts and scheduled-target previews identity-stable

Make SAM artifacts and scheduled-target previews identity-stable

Status: blocked
Tags: bug, backend, infra, testing, P0
Depends on: None
Blocks: #146
Next owner: HUMAN — authorized sandbox operator
Resume condition: Sanitized public issue evidence confirms preview PASS from exact commit 676a4f66cb4aafa2eef13287bc9c06e6c5f6e115 within the issue allowlist, never executed, and deleted after review.

Scope

Ship an independent architecture-preparation release that removes the two causes of the blocked #146 preview while preserving the normal application deployment path and all existing runtime identities.

The preparation release has two inseparable parts:

1. Make the shared SAM Lambda payload reproducible. Pin the build Node toolchain to exact Node 20.20.2 for local preview builds and both jobs in .github/workflows/deploy-dataops-v1.yml. Keep host/toolchain/cache provenance (including Node version, platform, architecture, and cache fingerprint) outside every deployed Lambda CodeUri payload. Preserve integrity and runtime-boundary verification using deterministic payload metadata or an external build sidecar under the owned .tmp/ cache boundary. A wrong Node version must fail before an artifact is accepted.
2. Replace only the six inline SAM Schedule events attached to the three #146 consumer functions with explicit AWS::Events::Rule and AWS::Lambda::Permission resources. Reuse the exact SAM-generated logical IDs so CloudFormation adopts the existing resources in place. Feed targets from three required, stable function-name parameters whose values are read from the existing stack outputs after OIDC credential setup; construct target ARNs from those names and the current partition/region/account. This removes rule/permission dependencies on mutable function resources while retaining the existing physical function, rule, and permission identities.

The six schedule pairs are:

• ConversationalExecutionWorkerFunctionExecutionRecovery and ...Permission
• ConversationalExecutionWorkerFunctionExecutionHealthPulse and ...Permission
• ConversationalResultDispatcherFunctionResultDelivery and ...Permission
• BackendFunctionDailyBackendCron and ...Permission
• BackendFunctionDailyBackendExport and ...Permission
• BackendFunctionDailyMailingExport and ...Permission

For each pair, preserve the currently transformed schedule expression, description, enabled/condition state, input, target ID, invocation principal/action, and source-rule relationship. Do not assign new physical names. The deployment workflow must resolve the existing ConversationalExecutionWorkerFunctionName, ConversationalResultDispatcherFunctionName, and BackendFunctionName outputs, validate that each is one non-empty Lambda name from the expected stack/account/region, and pass them as the three exact target-name parameters to the ordinary sam deploy command.

This issue must be implemented from the canonical deployed baseline that excludes commit 3491f65acad1986bd002cd1dabbb3e6b8eca0cfa. The current known deployed baseline is 6236865e509c0e142d364e6c56f7856d8f932076; if canonical main advances first, the orchestrator must prove the selected baseline still excludes #146. The local main ancestry is not authoritative while it contains unpushed descendants of 3491f65.

Acceptance Criteria

• One canonical file declares exact Node 20.20.2; local SAM builds and both workflow jobs use it, and the artifact build fails clearly on a different Node version before publishing or reusing an artifact.
• Host/toolchain/cache provenance is outside all six deployed Lambda payloads. No deployed file contains process.version, host platform/architecture, or a host-derived cache fingerprint.
• Two clean builds of the same source with the pinned toolchain produce byte-identical payload inventories and hashes for all six function artifacts; warm-cache reuse produces the same result.
• All six functions retain their current logical IDs, handlers, CodeUri: sam-build, runtime, configuration, roles/policies, and physical-name behavior. No alias, version, replacement function, live S3 code pointer, or packaged-template edit is introduced.
• Exactly the six listed SAM schedules become twelve explicit rule/permission resources under their existing transformed logical IDs. Their resolved behavior matches the pre-change transformed template except for the target reference changing to a stable function-name parameter.
• The three target-name parameters are required and have no environment-specific or physical-name default. After OIDC setup, the normal deploy job reads only the existing stack outputs, validates them, and passes exactly those values to sam deploy.
• A source-template diff is limited to adding the twelve explicit resources, removing their six inline event declarations, and adding the three target-name parameters. A processed-template diff proves zero logical-resource additions/removals/renames and no changes to physical names, retention/deletion policies, functions, IAM, queues, tables, buckets, alarms, Function URL, DynamoDB stream events, Sponsor send recovery, or the explicit Sponsor SES EventBridge rule beyond the approved one-time artifact and six schedule-reference migration.
• The existing schedule expressions, state conditions, event inputs, target IDs, and invocation permissions are covered by focused tests, including default-off/on transforms for the conversational schedules.
• The normal main GitHub Actions -> OIDC -> make sam-build -> artifact gates -> sam deploy --config-env full-sandbox path remains the only executing app deployment path. No manual app deployment, alternate stack, long-lived credential, or bypass is added.
• The candidate diff contains none of #146's provenance semantics: DataOpsEnvironment remains the deployed baseline shape (Default: prod with its existing allowed pattern), no explicit DataOpsEnvironment=sandbox deploy override is present, export/archive expectations and docs are unchanged, and none of #146's focused provenance tests are introduced. Shared files may change only for #217's packaging/schedule contracts.
• The branch contains no changes from unrelated local work or ../aws-infra, ../dtc-operations, ../datatasks, or ../podcast-assistant.
• [HUMAN] From the immutable accepted #217 commit, an authorized sandbox operator uses exact Node 20.20.2, the ordinary make sam-build/SAM packaging path, and --no-execute-changeset to create one preview. The preview is never executed and is deleted after review. Sanitized evidence must show zero adds/removes, no Replacement=True, and modifications limited to the reviewed one-time six-function payload normalization, the six listed rules, the six listed permissions (where Conditional is acceptable only for this reference migration after resolved old/new function names are proven equal), plus ordinary deployment-ID drift if present. Any other resource is BLOCK/ABORT.
• After preview PASS, the accepted commit is merged and pushed through normal main; On-Call records the terminal GitHub Actions/OIDC deployment result. There is no manual deployment or execution of the preview change set.
• [HUMAN] After the normal deployment, an authorized operator provides sanitized read-only CloudFormation/Lambda/EventBridge evidence that the same six functions, six rules, and six permissions retain their prior physical identities, all schedules retain their prior state/targets/input, and the stack is terminal-successful. No invocation, data write, export, provider message, or destructive operation is needed.
• #217 is closed only after the normal deployment and identity evidence pass. #146 remains unmerged/unshipped, is reconstructed from the deployed #217 baseline, and receives fresh Architecture, Security, Tester, PM, immutable preview, and release evidence.

Test Scenarios

Scenario: Host metadata cannot change deployed bytes

Given: Two clean caches and the exact same source tree under Node 20.20.2
When: All six SAM function artifacts are built twice and once through warm-cache reuse
Then: Canonical file inventories and SHA-256 hashes are identical, no payload contains host/cache provenance, and all artifact/runtime-boundary checks pass

Scenario: Wrong preview toolchain fails closed

Given: A local or CI SAM build running any Node version other than 20.20.2
When: The shared artifact builder starts
Then: It exits before accepting/reusing/publishing an artifact and names the required and actual versions

Scenario: Six schedules preserve behavior but lose function dependencies

Given: SAM-processed templates immediately before and after #217 with equivalent three target-name parameter values
When: Their resource graphs and resolved properties are compared
Then: The same six rule and six permission logical IDs remain, schedule behavior is unchanged, and none references Ref/GetAtt of the three mutable function resources

Scenario: Deployment resolves stable targets safely

Given: The existing full-sandbox stack and OIDC credentials in the normal deploy job
When: The workflow resolves the three existing function-name outputs
Then: It validates and passes the three names as required SAM parameters without writing configuration, editing a packaged template, or using live S3 code pointers

Scenario: Preparation candidate excludes #146

Given: The selected deployed baseline and the complete #217 candidate
When: The source, tests, docs, workflow, and processed-template diff are audited
Then: Only #217 packaging/schedule work is present and all sandbox provenance/export changes from 6236865...3491f65 are absent

Scenario: Independent release unblocks reconstruction

Given: Agent verification, PM acceptance, the HUMAN no-execute preview, normal OIDC deployment, and HUMAN read-only identity evidence all pass
When: #217 is closed and #146 is resumed
Then: #146 starts from the deployed #217 SHA and its fresh preview no longer changes unchanged function Code identities or the six schedule/permission resources

Proportional Verification

Agent-verifiable before commit (no AWS/provider access):

• Focused Node tests for artifact fingerprints/cache isolation, payload inventory, wrong-version failure, runtime boundary, and warm/cold reproducibility.
• Focused tests/infra/test_template.py tests for all twelve explicit resources, exact transformed behavior, stable target parameters, zero mutable-function references, baseline identity inventory, OIDC ordering, and #146-exclusion.
• Full uv run pytest tests/infra/test_template.py.
• Full backend tests, backend typecheck, and backend build because the shared packaged payload and its verifier change.
• make sam-validate, two clean make sam-build runs under Node 20.20.2, standalone SAM artifact/runtime/frontend gates, and a recorded six-artifact hash comparison under .tmp/.
• git diff/ancestry checks against the selected baseline and against 6236865...3491f65; no provider calls, preview, deploy, commit, push, or runtime invocation during implementation/testing/PM acceptance.

External evidence is deliberately separate:

• [HUMAN] One immutable non-executing CloudFormation preview, reviewed and deleted; never execute it.
• Normal merge/push and GitHub Actions OIDC deployment owned by the orchestrator/On-Call after preview PASS.
• [HUMAN] Sanitized read-only post-deployment identity evidence only; no functional invocation or data mutation is required.

Exact Sequencing

1. Orchestrator selects a clean canonical/deployed baseline that excludes 3491f65 and records its SHA.
2. Software Engineer implements only #217 in an isolated worktree; no commit, push, preview, provider access, or deployment.
3. Architecture reviews deterministic payload boundaries, the three stable target inputs, exact twelve-resource identity migration, and the preview allowlist.
4. Tester runs all agent-verifiable checks above and reports exact commands/hashes; PM accepts or rejects the immutable code slice.
5. After Tester PASS and PM acceptance, Software Engineer commits exactly the accepted paths with Refs #217; do not merge or push yet.
6. [HUMAN] Authorized operator creates/reviews/deletes the exact non-executing preview from that commit. Any result outside the allowlist is BLOCK/ABORT and returns to Architecture/Engineering.
7. After preview PASS, orchestrator merges and pushes main. The existing workflow performs the sole deployment through OIDC; On-Call monitors to terminal success.
8. [HUMAN] Authorized operator supplies sanitized read-only post-deployment identity evidence. PM closes #217 after all criteria pass.
9. Only then reconstruct/rebase #146 from the deployed #217 SHA and repeat its complete lifecycle with a fresh immutable preview. Do not reuse #146 commit 3491f65, its aborted preview, or its fingerprints as release evidence.

Safety Boundaries

• Preserve application stack name/config environment, logical and physical resource identities, existing OIDC role/path, parameter ordering/guards, and all data retention/deletion policies.
• Fail closed if a stack output is missing/ambiguous, a target differs from the existing function identity, the Node version differs, or a preview contains an unapproved resource/action.
• Do not execute a preparation preview, manually deploy the app, edit packaged templates, substitute deployed/live S3 pointers, create alternate resources, or introduce dual paths/compatibility fallbacks.
• Do not invoke Lambdas or schedules; write/export/import/restore/migrate data; access secrets/providers; send Telegram/sponsor/client messages; or publish private identifiers, ARNs, account IDs, paths, or credentials in this public issue.

Out of Scope

• Any #146 sandbox provenance, archive-key/manifest, DataOpsEnvironment, export test, or documentation change.
• Reconstructing, rebasing, accepting, previewing, merging, pushing, deploying, or closing #146.
• Changing function names, aliases, versions, handlers, runtimes, IAM, queues, tables, buckets, alarms, Sponsor schedules, the SES event rule, DynamoDB stream sources, or application behavior.
• Editing ../aws-infra or any source repository, changing OIDC/IAM/provider configuration, or adding a manual deployment route.
• Production, new environments, fresh-stack bootstrap design, runtime smoke/invocation, data mutation, export/import/restore, or private operational evidence.

Dependencies and Blocks

• Architecture diagnosis and evidence: https://github.com/DataTalksClub/dataops/issues/146#issuecomment-5461014165
• #217 has no implementation dependency and is intentionally releasable before #146.
• #146 remains blocked until #217 is deployed through normal main and its post-deployment identity evidence passes.

Start with .github/workflows/deploy-dataops-v1.yml, the SAM template, and tests/infra/test_template.py. Run make sam-validate and the focused infrastructure tests, then compare two clean make sam-build runs under Node 20.20.2. Done requires deterministic six-function artifacts, the six explicit rule/permission pairs, stable target parameters, and passing local verification without provider access or deployment.