posit-dev / posit-dev/images-shared

slack-build-notify: no persisted alert state — causes suppressed repeat-failure and recovery alerts

Open Beginner friendly
#677 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug cicd cvp:0 docker needs discussion observability tdp:2
Dominant language
Python
Stars
2
Forks
0
Avg merge
4d 13h
Merged PRs (30d)
22

Description

Summary

slack-build-notify re-derives "previous state" on every execution by picking whichever run was created immediately before the current run and reading its live conclusion — it never persists "what did we last tell Slack." This produces two related gaps:

  1. Repeat identical failures go unalerted. Only the first failure in a streak alerts; every subsequent identical failure compares failure == failure and is suppressed. This matches the action's documented intent ("notify on state transitions"), so it's arguably by design — but it means a persistent failure gets exactly one ping, ever.
  2. Recovery alerts are also suppressed when a fix comes via rerun. Rerunning a failed run's ci job to success re-executes the notify script within the same run, but the "previous run" lookup still excludes that run (it's "current") and lands on whatever was green before the original failure — so success == success and the recovery message never sends. This directly contradicts the action's stated purpose ("notifies... when a previously-failing workflow recovers"), so it's a clear bug, not a design tradeoff.

Evidence

Repeat-failure suppression — images-connect "Production", run https://github.com/posit-dev/images-connect/actions/runs/30186006522 alerted (first failure); images-connect "Development", runs https://github.com/posit-dev/images-connect/actions/runs/30295208545, https://github.com/posit-dev/images-connect/actions/runs/30289618868, https://github.com/posit-dev/images-connect/actions/runs/30285622599 all failed identically afterward and were each suppressed (No state transition (current: failure, previous: failure), skipping).

Recovery suppression — images-connect "Content Image", run https://github.com/posit-dev/images-connect/actions/runs/29672550578: CI job failed at attempt 1 (2026-07-19 04:05 UTC, alerted correctly), then succeeded on a rerun at attempt 2 (2026-07-19 18:10 UTC) — no recovery message was sent, since the lookup compared against the same prior run (2026-07-12, already green) both times.

Root cause

https://github.com/posit-dev/images-shared/blob/main/.github/actions/slack-build-notify/action.yml#L41-L86

previousRuns[0] is chosen by sorting completed runs by created_at and excluding the current run by ID — not by tracking what was last reported to Slack.

Proposed fix (bug: recovery suppressed on rerun)

When context.runAttempt > 1, compare against this run's own previous attempt (via GET /repos/{owner}/{repo}/actions/runs/{run_id}/attempts/{run_attempt - 1}) instead of the adjacent run. This directly fixes the confirmed rerun-recovery case, is small and self-contained, and doesn't touch repeat-failure behavior. It does not fix the compounding case where an intervening, never-rerun run masks a later new failure — that needs the redesign below.

For discussion — larger redesign (not proposed for immediate implementation)

The deeper issue is no persisted memory of "what did we last tell Slack." A more complete fix would track that explicitly (e.g., a repo variable updated on every notification sent), which would also let repeat failures get an optional periodic "still broken" reminder instead of permanent silence after the first alert, and would fix recovery detection regardless of reruns or interleaved runs. This needs an actual persistence mechanism (GitHub Actions has no built-in cross-run key-value store) and a product decision on whether reminder pings are wanted — flagging for discussion rather than proposing now.

Related

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .github/actions/slack-build-notify/action.yml around lines 41-86 and inspect how previousRuns[0] is selected. For rerun attempts, use the GitHub Actions run-attempt endpoint described in the issue and compare the current attempt with the prior attempt. Done means a failed run followed by a successful rerun sends the recovery notification.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions
Domain
ci-cd, devops
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.