Reconcile four retained failure queues after #166 returns the stack to steady state

Reconcile four retained failure queues after #166 returns the stack to steady state

Status: blocked — #166 Phase D, cleanup, and ordinary push-triggered OIDC deployment are complete, but no sanitized active-queue identity handoff or two-pass old-queue reconciliation has been posted; all currently possible work awaits explicit HUMAN/On-Call evidence
Tags: bug, infra, testing, P1
Depends on: #166 completed its repaired rollout and cleanup; now [HUMAN] On-Call/private binding of both active queue logical IDs, followed by the separately authorized [HUMAN] #137 read-only reconciliation
Blocks: None
Next owner: authorized HUMAN/On-Call owner to produce the private active-queue identity map and sanitized handoff; then the separately authorized HUMAN queue-reconciliation owner performs this issue's exact-name-only read checks
Resume condition: On-Call privately binds the final dataops-v1 stack and the two exact active queue logical IDs to their physical identities, gives that exact map to the authorized #137 HUMAN owner, and posts the bounded sanitized handoff with timestamps and the final #166 deployment reference; any missing, changed, transitional, or colliding identity returns to PM before old-queue checks
Evidence: initial retained-resource report in #128 comment 5137775335; current successful normal deployment run 31711997388 at 794076354048f6ff3417d55d13d334caf4237cea

Decision and current evidence

The issue remains open. The posted record proves:

• two failed July deployments retained four generated-name failure queues outside the restored stack;
• the initial read-only observation found all four at approximate available/in-flight/delayed 0/0/0, 14-day retention, no queue/redrive policy, no Lambda event-source mapping, and no current stack ownership;
• current source still declares retained 14-day active logical resources ConversationalExecutionFailureQueue and SponsorCommunicationFailureQueue, with the expected managed encryption choices;
• normal application deployment now succeeds. GitHub Actions run 31711997388 at exact pushed SHA 794076354048f6ff3417d55d13d334caf4237cea completed both Check DataOps v1 app and Deploy DataOps v1 app. Its stack deploy, runtime seeds, and deployed single-origin smoke all passed.

That workflow proves a terminal successful deployment but does not identify the active replacement queues and does not re-observe any old queue. No issue comment after the initial report records the two required visibility-timeout-separated checks. Therefore none of the old-queue reconciliation criteria is complete and closure is not authorized.

The repaired #166 rollout is the next application-stack mutation sequence. Its reviewed transforms are Tasks-table bounded and are expected to preserve all queue resources, but expectation is not evidence. #137 waits for the final post-#166 steady state so its active-versus-old identity comparison does not become stale immediately after collection.

Exact retained inventory

Only these four already-public exact physical queue names are in scope:

• dataops-v1-ConversationalExecutionFailureQueue-5tIXadG96UPw
• dataops-v1-ConversationalExecutionFailureQueue-uaUHwrRVO3KW
• dataops-v1-SponsorCommunicationFailureQueue-8iVYd6AmknFS
• dataops-v1-SponsorCommunicationFailureQueue-wtudxiKwXhgF

Never resolve a prefix, glob, search result, inferred suffix, or newly discovered queue under this issue. An exact name that no longer resolves is recorded as not found with timestamp and identity method; it is not recreated and no broader search is authorized.

This issue authorizes evidence collection only. It does not authorize receive, peek, body inspection, purge, redrive, detach, policy/tag/encryption change, retention change, deletion, recreation, or any other queue/AWS mutation.

Ownership and sequencing

#166 operator and On-Call

#166 owns only its reviewed Tasks cutover and deployment lifecycle. It must not inspect, clean up, delete, purge, redrive, retag, or otherwise mutate these old queues on behalf of #137.

After repaired Phase D and cleanup:

1. record the exact final pushed/dispatched SHA and successful workflow/run;
2. prove the dataops-v1 stack is terminal UPDATE_COMPLETE and ordinary push/OIDC deployment is restored;
3. privately resolve only the active logical resources:
   • ConversationalExecutionFailureQueue;
   • SponsorCommunicationFailureQueue;
4. bind each logical resource to current physical name, ARN, URL, creation timestamp, stack ID, and relevant output without posting account ID or raw URL publicly;
5. give the authorized #137 HUMAN owner that private exact map. The public handoff contains only logical IDs, collision-resistant identity hashes, timestamps, and the terminal run reference.

If either logical resource is absent, replaced unexpectedly, transitional, or changed by a #166 change set/template, #137 remains blocked and returns to PM. Do not continue by guessing which queue is active.

Authorized #137 HUMAN reconciliation owner

Using the final active map and four literal old names:

1. resolve each old queue by its exact literal name;
2. bind the resolved queue privately to exact URL/ARN and creation timestamp;
3. compare exact old identity with both active identities;
4. collect the two separated read-only observations below;
5. publish one sanitized report and stop.

On-Call or PM reviews the sanitized report against every criterion. Neither reviewer performs cleanup under this issue.

Required read-only evidence

For each exact old queue that still resolves, observation A and observation B must be separated by at least that queue's returned VisibilityTimeout. Record both UTC timestamps and the returned timeout. A longer delay is allowed; a shorter delay fails the gate.

Each observation must prove through read-only APIs/configuration:

• ApproximateNumberOfMessages, ApproximateNumberOfMessagesNotVisible, and ApproximateNumberOfMessagesDelayed are exactly 0/0/0;
• MessageRetentionPeriod remains 1209600 seconds;
• encryption remains SQS-managed for the conversational queues and alias/aws/sqs for Sponsor queues;
• Policy, RedrivePolicy, and RedriveAllowPolicy are absent;
• the queue ARN/creation timestamp remain the same between A and B;
• neither active queue identity equals the old ARN/name/URL identity;
• no current CloudFormation stack resource/output/processed-template reference points to the old identity;
• no Lambda event-source mapping destination, Lambda asynchronous failure destination, EventBridge Pipe/target, SNS subscription, or active application configuration points to the old ARN/URL/name;
• no active queue redrive source/allow relationship points to or from the old queue.

Use exact-resource reads. Do not list broad queue prefixes or record raw policy/tag/config payloads publicly. If a necessary relationship API cannot be bounded to exact known identities, stop and PM-regroom rather than widen discovery.

If an old exact name returns QueueDoesNotExist, record two exact-name resolutions separated by the former/default visibility timeout only if the reviewer accepts that method as sufficient absence evidence. Do not infer deletion from one not-found response or enumerate similar queues.

Acceptance criteria

• A normal main deployment has terminally succeeded after the July failures: run 31711997388 at 794076354048f6ff3417d55d13d334caf4237cea passed stack deploy, seeds, and deployed smoke.
• [HUMAN] #166 completed its repaired A/B/C/final-preflight/D/cleanup sequence; the final stack reached UPDATE_COMPLETE and ordinary push-triggered OIDC deployment was restored (final run 31747348640, attempt 2).
• [HUMAN] The final stack privately binds both exact active logical IDs to physical name/ARN/URL/creation timestamp/stack ID; sanitized hashes and deployment reference are posted, and neither active identity equals an old queue.
• [HUMAN] Each of the four literal old names is resolved without prefix/glob discovery and is privately bound to exact ARN/URL plus creation timestamp, or receives the accepted exact-name two-pass not-found treatment.
• [HUMAN] For every still-resolving old queue, two observations separated by at least its returned VisibilityTimeout show exact available/in-flight/delayed 0/0/0 and stable identity.
• [HUMAN] Both observations prove absent queue policy, redrive policy/allow relationship, Lambda mapping/destination, EventBridge Pipe/target, SNS subscription, CloudFormation reference, output, processed-template reference, and active application reference.
• [HUMAN] Retention/encryption match the recorded contract and the public report exposes no URL, account ID, policy, tag, message body, private payload, or credentials-adjacent metadata.
• [HUMAN] A reviewer accepts one sanitized reconciliation report containing exact UTC timestamps, timeout separation, logical/identity hashes, exact check categories, #166/final deployment reference, anomalies, and verdict.
• Stop after evidence/review. Any cleanup or deletion requires a new explicitly authorized issue with an exact-resource rollback/deletion plan; #137 grants none.

Failure handling

Any nonzero count, changing identity, active/old collision, missing active replacement, non-terminal stack, #166 queue-resource change, policy/redrive/reference/mapping/target/subscription, ambiguous lookup, insufficient timeout separation, unbounded discovery requirement, or inconsistent observation leaves #137 blocked.

Do not make a check pass by receiving, purging, redriving, detaching, editing policy, changing encryption/retention/tags, deleting, or recreating anything.

Sanitized report schema

The public report records only:

• #166 final and cleanup workflow URLs, exact source SHA, terminal conclusions, and stack status;
• active logical IDs with SHA-256 hashes of privately retained exact physical identity tuples;
• each old public name with SHA-256 ARN/URL identity hashes and creation timestamp, or exact-name not-found status;
• observation A/B UTC timestamps and returned visibility timeout;
• 0/0/0 counts;
• pass/fail booleans for retention, encryption, policy, redrive, Lambda, EventBridge, SNS, CloudFormation, outputs/template, application reference, and active-identity mismatch;
• reviewer verdict and any blocking category.

Do not post raw queue URLs, account IDs, ARNs containing account IDs, message bodies, policies, tags, private configuration, or credentials.

Out of scope

• Any AWS call before the explicit post-#166 HUMAN handoff.
• Queue list/prefix/glob discovery, message receive/peek/body inspection, purge, redrive, detach, tag/policy/encryption/retention change, deletion, recreation, or cleanup.
• Application or #166 deployment work, Tasks/Cards tables, Sponsor GSI work, IAM changes, provider/Telegram calls, credentials, data/import/export/restore, repository edits/tests, or any resource not one of the four literal old names and two active logical IDs.

Wait for the post-#166 HUMAN handoff and review the exact active logical IDs plus the four literal queue names listed in the issue. Then perform only the authorized exact-name, read-only checks, including two observations separated by each visibility timeout, and publish the specified sanitized reconciliation report; no repository files or tests are in scope.