BrighterCommand / BrighterCommand/Brighter
aws-cleanup sweep has never completed since #4298: killed by timeout-minutes every run, and reported as 'cancelled'
- Dominant language
- C#
- Stars
- 2.5k
- Forks
- 296
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 21
Description
## Summary
The `AWS Test Resource Cleanup` workflow has **not completed successfully once** since #4298 merged. Every
run since hits `timeout-minutes: 30` and dies part-way through the queue sweep. Because GitHub reports a
job killed by `timeout-minutes` as **`cancelled`**, not `timed_out` or `failure`, twelve consecutive dead
sweeps produced no failure notification and went unnoticed.
The consequence was a deadlock: the sweep is the mechanism that clears a leak backlog, and it could never
get far enough to clear one.
## Evidence
Every run from `2026-09-07T20:47Z` onward is `cancelled` after ~30 minutes:
| run | trigger | created → updated | duration |
|---|---|---|---|
| [`34194189605`](https://github.com/BrighterCommand/Brighter/actions/runs/34194189605) | schedule | 06:19:50 → 06:50:10 | 30m20s |
| [`34185387744`](https://github.com/BrighterCommand/Brighter/actions/runs/34185387744) | workflow_run | 03:59:26 → 04:29:48 | 30m22s |
| [`34173620862`](https://github.com/BrighterCommand/Brighter/actions/runs/34173620862) | schedule | 00:32:00 → 01:02:21 | 30m21s |
| [`34168461278`](https://github.com/BrighterCommand/Brighter/actions/runs/34168461278) | workflow_run | 22:58:28 → 23:28:48 | 30m20s |
| … 8 more, all ~30m00s–30m30s, all `cancelled` | | | |
The break is sharp and lines up exactly with #4298 (`0c4bd3344`, merged `2026-09-07T20:30Z`):
```
success 2026-09-07T20:29:54Z -> 2026-09-07T20:32:06Z (2m12s) <- last run before #4298
cancelled 2026-09-07T20:47:05Z -> 2026-09-07T21:18:11Z (31m06s) <- first run after
```
Before #4298 every run finished in 1–3 minutes; after it, none finishes at all.
## Why #4298 changed it — this is a consequence of the fix working, not a regression in it
#4298 added `GENERATED_TEST_PATTERN` to `clean_failed_tests_aws_assets.sh`:
```bash
GENERATED_TEST_PATTERN="(sqs|sns)-(std|fifo)(-ch)?-[0-9a-f]{32}"
TEST_NAME_PATTERN="^($TEST_PREFIXES|$GENERATED_TEST_PATTERN)"
```
That is precisely the fix its diagnosis called for — before it, the leaked generated-test resources matched
neither the `Environment=Test` tag query nor the old hand-written prefixes, which is why ~26k orphans
accumulated invisibly. The same PR raised the timeout `5 → 30` in anticipation.
**30 minutes is not enough for the backlog the new pattern can now see.** From the workflow's own diagnostic
step:
| sweep | topics | FIFO topics | queues |
|---|---|---|---|
| 22:34 | 642 | 0 | 7,330 |
| 22:58 | **0** | 0 | 7,330 |
| 00:32 | **0** | 0 | 7,284 |
Topics clear because they are swept first. Queues moved **7,330 → 7,284 in two 30-minute runs — about 46
per run.** At that rate the backlog would never have cleared.
The cost is per-queue API calls: `partition_by_age` issues one `sqs get-queue-attributes` per matched queue
to read `CreatedTimestamp`, then one `delete-queue` each — ~14,400 calls for this backlog, at
`CLEANUP_PARALLELISM` (default 16).
## Confirmed by running it by hand
Run from a developer machine against `eu-west-1` with no 30-minute cap:
- `--dry-run`: **7,192 test queues matched, 7,156 actionable**, 36 correctly deferred as too young (an
`aws-ci` job was in flight; the `CLEANUP_MIN_AGE_SECONDS=3600` guard worked exactly as designed).
- Real run: completed in **~1 hour**. A second pass matched **0**.
- Account now reads **0 SQS queues, 0 SNS topics, 0 `Environment=Test` resources**.
So the script is correct and effective — it is only ever being killed.
## Suggested fixes
Roughly in order of effort:
1. **Raise `timeout-minutes`.** Simplest, but it only moves the cliff; the sweep's runtime scales with the
backlog, and the failure mode on hitting it again is silent.
2. **Make the sweep bounded and resumable** — give it a deletion budget per run and let it exit `success`
having made progress, rather than being killed mid-flight. Turns an invisible timeout into visible,
incremental work.
3. **Cut the per-queue age call.** The `get-queue-attributes` round trip doubles the API cost of every
queue. The guard only needs to protect *recent* resources, so a cheaper shape may exist.
4. **Make a killed sweep visible.** A step that reports the remaining resource count at exit, or an
explicit deadline check that fails loudly, would have surfaced this on day one instead of after twelve
silent runs.
## ⚠️ Secondary: the name sweep discards the AWS error, so its log cannot be trusted
Of the 7,156 deletes in the manual run, **1,845 printed `WARNING: failed to delete queue …` and had in fact
succeeded** — the account ended up empty. `clean_failed_tests_aws_assets.sh:522-527`:
```bash
if aws sqs delete-queue --queue-url "$1" >/dev/null 2>&1; then
echo " Deleted untagged test queue: ${1##*/}"
else
echo " WARNING: failed to delete queue ${1##*/}"
fi
```
`2>&1` to `/dev/null` throws away the reason, so a lost race between the 16 parallel workers (or a
throttle that the SDK would have retried) is indistinguishable from a genuine failure. Keeping the error
text would make the log answer the question you actually go to it with.
**Also worth knowing when reading these logs:** `sqs list-queues` is eventually consistent and keeps
listing deleted queues for a while — a census taken immediately after the sweep showed 70 survivors, and
the same query minutes later showed 0. Do not diagnose a survivor from a single listing.
## Related
- #4298 — added the name sweep this issue is a consequence of
- #4240 — the conformance suite whose `aws-ci` job was blocked by the resulting quota exhaustion
Contributor guide
Assessment
This issue has not been assessed yet.