Move the canonical SAM app template to aws-infra without moving app deployment

Move the canonical SAM app template to aws-infra without moving app deployment

Status: blocked — #166 is complete; await #217 terminal closeout before refreshing the canonical template
Tags: enhancement, infra, testing, P1
Depends on: #217
Blocks: None
Next owner: HUMAN authorized operator on #217; then Product Manager for #173 reconciliation

Current prepared state

• The existing aws-infra clone still contains commit 5dd8a1a0d118ed116c1833538a499fbfde46b51b; no current local or remote issue-173 branch points to it. The commit is inert preparation only and is not an approved DataOps pin.
• The prepared template was byte-identical when copied but is stale versus current origin/main, and accepted #217 changes the same template and deploy workflow without yet reaching origin/main. Refresh only from the exact post-#217 deployed source; do not merge or pin 5dd8a1a.
• #185 is Tester/PM accepted and committed in DataOps as 247ea7cd1a83db0018418276f138218f372a9b1c. Its isolated, content-addressed Lambda builder is now a required accepted ancestor of #173. The cross-repository template resolver must feed SAM without reintroducing root dependency mutation, a checkout install in the deploy job, or a second build.
• Both shared checkouts contain unrelated work. Final implementation remains isolated-worktree work with explicit paths; no cleanup, checkout, stash, or commit of another session’s files is authorized.

Scope

Move the only tracked application-stack template from dataops/infra/template.full.yaml to aws-infra/sandbox/dataops/template.application.yaml. aws-infra owns the canonical CloudFormation/SAM definition, but it does not gain an application deployment workflow and a push there does not touch AWS.

DataTalksClub/dataops remains the sole app build and deploy authority. Its existing main-branch GitHub Actions workflow checks out a committed, immutable aws-infra revision, builds the DataOps source against that exact template, verifies the artifact, and deploys .aws-sam/build/template.yaml through the existing GitHub Actions OIDC role.

The concrete boundary is:

1. Add a single 40-character aws-infra commit pin to DataOps (for example .github/aws-infra.ref). Never follow main, a tag, or another mutable ref during a deployment.
2. Because DataTalksClub/aws-infra is private, both DataOps workflow jobs use a dedicated repository secret named AWS_INFRA_READ_TOKEN: a fine-grained token limited to DataTalksClub/aws-infra with repository Contents read-only. The checkout uses the committed SHA and persist-credentials: false. Do not reuse the runtime knowledge token and do not add AWS keys.
3. A DataOps-owned resolver verifies the pin format and materializes the pinned sandbox/dataops/template.application.yaml git blob under ignored .tmp/. CI checks out that exact revision first; local development may read the same git object from the sibling ../aws-infra clone without checking out, stashing, or modifying its current branch. The materialized file must be byte-identical to the pinned blob.
4. The moved template declares each Lambda CodeUri relative to the DataOps build base (infra/sam-build). make sam-build passes both the resolved template path and --base-dir set to the DataOps repository root, so the tiny makefile adapter continues to invoke the just-checked-out DataOps bundler. The accepted #185 builder remains the sole content-addressed Lambda build: it treats the DataOps checkout as immutable input, installs/builds only inside its fingerprint-owned workspace, emits six independent artifacts, and may use a validated warm cache. The move must not package code from aws-infra or another DataOps revision, add a deploy-job install, or add a second SAM/backend build.
5. samconfig.toml no longer names the removed local source template. The deploy configuration continues to deploy only .aws-sam/build/template.yaml, which is the transformed output from the exact template/code pair just validated and built.

Acceptance Criteria

• aws-infra/sandbox/dataops/template.application.yaml is the sole tracked canonical application-stack template; dataops/infra/template.full.yaml is deleted and no generated or fallback copy is committed to DataOps.
• DataOps records one immutable 40-character aws-infra commit SHA, and validation fails before build/AWS credential exchange when the pin is malformed, unavailable, or does not contain the expected template path.
• The pinned revision is checked out read-only in both the checks and deploy jobs using AWS_INFRA_READ_TOKEN and persist-credentials: false; neither job follows an aws-infra branch head.
• [HUMAN] AWS_INFRA_READ_TOKEN exists in the DataOps repository and is limited to Contents read-only on the single private DataTalksClub/aws-infra repository.
• Local resolution reads the pinned git object from ../aws-infra without switching branches or changing/stashing unrelated dirty files; CI materializes the same bytes under DataOps .tmp/.
• SAM validation and the one shared esbuild build consume the same resolved template bytes. All six functions resolve infra/sam-build against the DataOps root and retain their declared bundled handlers.
• The paired DataOps release retains #185 commit 247ea7cd1a83db0018418276f138218f372a9b1c in ancestry and preserves its isolated checkout-safe cold build, validated warm cache, six independent artifacts, one deploy SAM build, and no deploy-job npm ci.
• The deploy step still consumes .aws-sam/build/template.yaml, uses the existing dataops-github-actions-deploy OIDC role, and contains no static AWS credentials or manual app-deploy path.
• A push to aws-infra alone cannot trigger or perform a DataOps app deployment. A template change becomes deployable only after DataOps deliberately updates its pin and pushes that reviewed commit to main.
• Template-path tests, workflow contract tests, samconfig.toml, AGENTS.md, docs/local-development.md, the runtime architecture docs, and aws-infra/sandbox/dataops/README.md describe and enforce the new ownership/pin/build boundary with no stale local-template instructions.
• The DataOps infra/planning tests load the pinned canonical template rather than a fixture copy, and the aws-infra test suite covers the new canonical file at its owning path.
• make sam-validate, make sam-build, all packaged-artifact gates, backend tests/typecheck/build, DataOps infra/planning tests, and the affected aws-infra tests pass from clean worktrees.
• After the DataOps pin/deletion commit reaches main, On-Call evidence shows the existing OIDC workflow deployed the expected DataOps SHA plus pinned aws-infra SHA, CloudFormation reached UPDATE_COMPLETE, and production smoke passed.

Test Scenarios

Scenario: exact pinned cross-repository build

Given DataOps records an aws-infra SHA whose tree contains the canonical template
When the checks and deploy jobs run
Then each fetches that exact private commit, validates identical template bytes, and the deploy job packages the current DataOps source once against it before deploying the built template.

Scenario: app code changes without an infrastructure change

Given the pin is unchanged
When a DataOps application commit reaches main
Then the normal DataOps workflow builds and deploys that new app code using the same pinned template and existing OIDC role.

Scenario: infrastructure changes do not self-deploy

Given a new application-template commit reaches aws-infra/main
When no DataOps pin update is pushed
Then no DataOps workflow runs and no application stack changes.

Scenario: invalid or unavailable pin fails closed

Given the committed pin is malformed, missing from the private remote, or lacks sandbox/dataops/template.application.yaml
When validation begins
Then the workflow fails before SAM build and before AWS credential exchange; it never falls back to a branch head or a local DataOps template.

Scenario: local sibling repo has unrelated work

Given ../aws-infra is on another branch with unrelated modified/untracked files
When a developer runs the resolver and SAM checks
Then it reads the template blob at the pinned commit without checking out, cleaning, stashing, or editing that working tree.

Scenario: rollback a paired release

Given a newly pinned template/code pair fails deployment or smoke verification
When the DataOps release commit is reverted
Then the revert restores the previous compatible app code and aws-infra pin together, and the ordinary DataOps main-branch OIDC workflow redeploys that known-good pair. aws-infra history is not rewritten and no manual SAM deploy is introduced.

Cross-repository commit and release sequence

1. Complete #166 first so the final card/task schema is canonical before the template moves.
2. Create clean isolated worktrees for both repositories. The current shared DataOps and aws-infra checkouts contain unrelated work; do not clean, stash, overwrite, stage, or commit it.
3. After #166 cleanup, in the clean aws-infra worktree refresh the prepared template.application.yaml from the exact final DataOps source, add ownership documentation and owning tests, and obtain lifecycle review before committing and pushing it. Do not use the stale 5dd8a1a snapshot as the pin. The refreshed aws-infra commit is inert because aws-infra has no app deploy workflow.
4. Confirm the read-only cross-repository token is configured, then in the clean DataOps worktree record that exact already-pushed SHA, delete the local template, and update the resolver, Make/SAM/workflow paths, tests, and docs in one reviewed release increment.
5. Push DataOps main. Only this push triggers the normal OIDC app deployment. Record both repository SHAs in the issue/on-call evidence.
6. For every later template change, repeat the same order: push reviewed aws-infra template commit, then update the immutable pin in a reviewed DataOps commit. Coordinated app/template changes keep the code and pin in the same DataOps release commit.

Out of Scope

• Giving aws-infra CI/CD authority to deploy the DataOps application.
• Manual application deployment or changing the existing GitHub Actions OIDC AWS trust model.
• Copying or vendoring a tracked template back into DataOps, following aws-infra/main, or adding a fallback template.
• Reusing the content/knowledge GitHub token or granting write/admin/workflow access to the cross-repository checkout token.
• Changing Lambda behavior, events, IAM policies, environment variables, tables, indexes, persisted data, or resource names as part of the file move.
• Moving the DataOps bundler or infra/sam-build packaging adapter into aws-infra.
• Cleaning or committing either repository's unrelated existing worktree changes.
• Publicizing aws-infra or creating an aws-infra application-deploy workflow.

Dependencies

• #172 is complete and provides the single shared esbuild artifact plus the small infra/sam-build adapter.
• #185 is satisfied by accepted DataOps commit 247ea7cd1a83db0018418276f138218f372a9b1c. The final #173 DataOps commit must retain that exact commit in ancestry and preserve its isolated cold build, validated warm cache, six independent artifacts, one deploy SAM build, and no deploy-job npm ci.
• #166 is fully satisfied and closed. Its terminal ledger proves all cutover phases, cleanup, ordinary main-push OIDC restoration, production verification, and maintenance-window closure; current origin/main is marker-free and declares the final Tasks schema.
• #182 was satisfied inside #166 and is not a remaining dependency.
• #217 is the active sequencing dependency because its accepted, unshipped commit changes both the canonical template and ordinary deploy workflow. Moving ownership first would invalidate #217’s exact preview/release boundary and refresh #173 from superseded source.
• Engineering resume condition: #217 is closed after its accepted commit passes the authorized no-execute preview, ordinary DataOps main deployment, and sanitized post-deployment identity review, and that accepted state is present on origin/main. Then refresh and review the canonical application template from that exact source.
• Release resume condition: the refreshed, reviewed aws-infra template commit is pushed, and a repository administrator confirms the dedicated DataOps AWS_INFRA_READ_TOKEN secret uses a fine-grained token limited to Contents read-only on only DataTalksClub/aws-infra and is available to both jobs. This is #173’s sole HUMAN credential prerequisite and grants no AWS access. Only then may the paired DataOps pin/deletion commit be pushed.

First wait for the #217 terminal closeout, then work only in clean isolated worktrees. Read dataops/infra/template.full.yaml, .github/aws-infra.ref, infra/sam-build, samconfig.toml, and the existing workflow and tests; compare them with aws-infra/sandbox/dataops/template.application.yaml. Done means the pinned template is resolved identically in checks and deploy, the old copy is removed, documentation and tests enforce the boundary, and all listed build, test, and deployment gates pass.