posit-dev / posit-dev/images-shared
slack-build-notify: no persisted alert state — causes suppressed repeat-failure and recovery alerts
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2
- Forks
- 0
- Avg merge
- 4d 13h
- Merged PRs (30d)
- 22
Description
Summary
slack-build-notify re-derives "previous state" on every execution by picking whichever run was created immediately before the current run and reading its live conclusion — it never persists "what did we last tell Slack." This produces two related gaps:
- Repeat identical failures go unalerted. Only the first failure in a streak alerts; every subsequent identical failure compares
failure == failureand is suppressed. This matches the action's documented intent ("notify on state transitions"), so it's arguably by design — but it means a persistent failure gets exactly one ping, ever. - Recovery alerts are also suppressed when a fix comes via rerun. Rerunning a failed run's
cijob to success re-executes the notify script within the same run, but the "previous run" lookup still excludes that run (it's "current") and lands on whatever was green before the original failure — sosuccess == successand the recovery message never sends. This directly contradicts the action's stated purpose ("notifies... when a previously-failing workflow recovers"), so it's a clear bug, not a design tradeoff.
Evidence
Repeat-failure suppression — images-connect "Production", run https://github.com/posit-dev/images-connect/actions/runs/30186006522 alerted (first failure); images-connect "Development", runs https://github.com/posit-dev/images-connect/actions/runs/30295208545, https://github.com/posit-dev/images-connect/actions/runs/30289618868, https://github.com/posit-dev/images-connect/actions/runs/30285622599 all failed identically afterward and were each suppressed (No state transition (current: failure, previous: failure), skipping).
Recovery suppression — images-connect "Content Image", run https://github.com/posit-dev/images-connect/actions/runs/29672550578: CI job failed at attempt 1 (2026-07-19 04:05 UTC, alerted correctly), then succeeded on a rerun at attempt 2 (2026-07-19 18:10 UTC) — no recovery message was sent, since the lookup compared against the same prior run (2026-07-12, already green) both times.
Root cause
previousRuns[0] is chosen by sorting completed runs by created_at and excluding the current run by ID — not by tracking what was last reported to Slack.
Proposed fix (bug: recovery suppressed on rerun)
When context.runAttempt > 1, compare against this run's own previous attempt (via GET /repos/{owner}/{repo}/actions/runs/{run_id}/attempts/{run_attempt - 1}) instead of the adjacent run. This directly fixes the confirmed rerun-recovery case, is small and self-contained, and doesn't touch repeat-failure behavior. It does not fix the compounding case where an intervening, never-rerun run masks a later new failure — that needs the redesign below.
For discussion — larger redesign (not proposed for immediate implementation)
The deeper issue is no persisted memory of "what did we last tell Slack." A more complete fix would track that explicitly (e.g., a repo variable updated on every notification sent), which would also let repeat failures get an optional periodic "still broken" reminder instead of permanent silence after the first alert, and would fix recovery detection regardless of reruns or interleaved runs. This needs an actual persistence mechanism (GitHub Actions has no built-in cross-run key-value store) and a product decision on whether reminder pings are wanted — flagging for discussion rather than proposing now.
Related
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with .github/actions/slack-build-notify/action.yml around lines 41-86 and inspect how previousRuns[0] is selected. For rerun attempts, use the GitHub Actions run-attempt endpoint described in the issue and compare the current attempt with the prior attempt. Done means a failed run followed by a successful rerun sends the recovery notification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions
- Domain
- ci-cd, devops
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100